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Description 

Technical Field 

5 [0001] This invention relates to a key signal generating apparatus and a picture synthesis apparatus, and a key 
signal generating method and a picture synthesis method. More particularly, this invention relates to a key signal gen- 
erating apparatus and a picture synthesis apparatus, and a key signal generating method and a picture synthesis 
method which are adapted for detecting change of pixel value between pixels constituting pictures to generate a key 
signal on the basis of the change of the pixel value, thereby making it possible to obtain an accurate key signal. 

10 

Background Art 

[0002] For example, in movies and other image productions, in the case where various simulations of pictures (im- 
ages) are carried out, or any special effect is given to pictures, a remarked portion (object) is extracted from picture, 
15 or the extracted portion is synthesized with respect other pictures. Such a partial extraction from picture and/or synthesis 
(of the extracted portion) is carried out by using key signal. 

[0003] As a technology relating to such a key signal, an example thereof is proposed in the specification and the 
drawings of the U.S. Patent No. 4,488,169 (Registered Date: December 11, 1984) by the applicant of this invention. 
[0004] As the key signal, there are a key signal called hard key and a key signal called soft key. Now, in the case 

20 where, e.g., the remarked portion within picture used for generating the key signal is caused to be the foreground 
(foreground picture), and the portion except for the above is caused to be the background (background picture), a 
binary key signal (mask picture) in which the range of the foreground in the picture is caused (assumed) to be 1 and 
the range of the background is caused (assumed) to be 0 as shown in FIG. 1 A is called hard key. On the contrary, a 
key signal which can take not only two values of 0 and 1 , but also continuous real numbers within the range from 0 to 

25 1 as shown in FIG. 1 B in consideration of alias and/or motion blur which will be described later taking place in picture 
is called soft key. Here, the area where the key signal takes values within the range from 0 to 1 is called gray area as 
the occasion may demand. 

[0005] Namely, it can be said that the hard key is a signal having a sharp boundary (border) line (signal in which 
inclination (gradient) of the boundary line where the key signal changes from 0 (or 1 ) to 1 (or 0) is sharp), and the soft 
30 key is a signal having a smooth (gentle) boundary (border) line (signal in which inclination (gradient) of the boundary 
line where the key signal changes from 0 (or 1) to 1 (or 0) is smooth (gentle)). 

[0006] it is now assumed that a (a value) shown in FIGS. 1 A and 1 B indicates continuous key signal values and also 
indicates, in this case, values of the key signal of pixel units at the same time. In practice, e.g., continuous key signals 
are caused to undergo filtering by the range of pixel, etc. to use, as the key signal of the pixel unit, the result by that 
35 filtering at the center of corresponding pixel. In this case, in the picture, pixel value of the pixels at the boundary portion 
between the foreground and the background is a value in which respective signal components of the foreground and 
the background are interposed. From facts as above, it can be said that a indicates contribution ratio (factor) indicating 
to what degree the foreground contributes with respect to respective pixels. 

[0007] Meanwhile, it is rare that in the case where both the foreground and the background are stationary, the bound- 

40 ary between the foreground and the background is positioned at the boundary line between pixels constituting a cor- 
responding picture. Ordinarily, as shown in FIG. 2A, the boundary between the foreground and the background exists 
within the pixel having finite dimensions. However; since the pixel is the minimum unit constituting the picture, even if 
the boundary between the foreground and the background as described above exists within the pixel, that pixel is 
caused to have color or brightness (pixel value) of any one of the foreground and the background as shown in FIG. 

45 2B. For this reason, the boundary line between the foreground and the background is drawn along the boundary line 
between pixels as shown in FIG. 2C. When the entirety of such a picture is viewed, unnatural flicker, i.e., alias takes 
place at the portion of the boundary between the foreground and the background (boundary line between pixels). 
[0008] Accordingly, even if a key signal smoothly indicating the boundary line between the foreground and the back- 
ground can be obtained, alias would take place in the synthetic picture obtained as the result of the fact that the key 

so signal mentioned above is used to carry out picture synthesis. 

[0009] In view of the above, as a method of preventing alias taking place in the synthetic picture, there are, e.g., 
methods disclosed in the Japanese Patent Application Laid Open No. 232787/1985 and the Japanese Patent Appli- 
cation Laid Open No. 340671/1 992. In the case of the method disclosed in the Japanese Patent Application Laid Open 
No. 232787/1985, the hard key is used to carry out picture synthesis, and to apply low-pass filter (LPF) only to the 

55 boundary portion between the foreground and the background to allow the boundary portion thereof to be blurred to 
allow it to be smooth to look at. In this method, since the hard key is used, but LPF is applied to the boundary portion 
between the foreground and the background, it can be said that picture synthesis is carried out by using key in which 
the key signal is caused to have gradient (inclination), i.e., substantially soft key. 
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[0010] Moreover, in the method disclosed in the Japanese Patent Application Laid Open No. 340671/1992, a block 
in which a remarked pixel on the boundary between the foreground and the background is caused to be the center is 
set to calculate a key signal (soft key) from the pixel pattern constituting the foreground and the background within that 
block to carry out, by using such a key signal, extraction of the foreground, and synthesis of the foreground and the 

5 background to thereby reduce the alias. 

[0011] However, since the soft key used in the above-mentioned method is such that value of a at the inclined 
(gradient) portion (the portion of 0 < a < 1 in FIG. (B)), i.e., in the gray area is uniform, there was the problem that it is 
difficult to cope with motion blur taking place, e.g., in the case where the foreground is moving. 
[0012] The motion blur will now be briefly described. The moving picture is realized as the result of the fact that 

10 picture (still picture) is continuously displayed, e.g., in frame units, etc. For example, assuming now that a circular 
object moving from the left direction to the right direction within picture exists, in the case where the frame period is 
assumed to be a short time such that it can be considered to be infinitesimal (infinitely small) with respect to the 
movement velocity of that object, that object is displayed as a circular object in respective frames as shown in FIG. 
3A. However, in the case where the frame period is not a short time such that it can be considered to be infinitesimal 

15 with respect to the movement velocity of the circular object, picture of frame at a certain time t 2 becomes a picture in 
which movement of the object from time t 1 of frame preceding by one to the time t 2 is reflected as shown in FIG. 3B. 
Namely, the object within the picture of the frame at the time t 2 is not circular picture image, but picture image extended 
(elongated) in a thin manner in the direction of that motion such that the contour is blur. Such a phenomenon is called 
motion blur. 

20 [0013] It is to be noted that while motion blur takes place not only in the case where any one of the foreground and 
the background is moving, but also in the case where both are independently moving, in the case where only the 
background is moving and in the case where both are independently moving, it is possible to consider those cases 
similarly to the case where only the foreground is moving if the background is taken as reference. In addition, in the 
case where motion blur takes place, clear boundary between the foreground and the background does not exist. 

25 [0014] As a method of generating a key signal in which motion blur is taken into consideration as described above, 
there are methods, e.g., the Japanese Patent Application Laid Open No. 153493/1993 and the Japanese Patent Ap- 
plication Laid Open No. 236347/1993. In the method disclosed in the Japanese Patent Application Laid Open No. 
153493/1 993, current frame and frame preceding (earlier) by one are compared with each other to thereby determine 
sum total with respect to the entirety of frame of change quantities (magnitudes) of pixel value of pixels constituting 

30 the foreground of the current frame (or the frame preceding by one) and pixel value corresponding to pixels of the 
frame preceding by one (or the current frame) to determine the gray area of the soft key from the sum total value. 
However, in this method, since the soft key is determined in dependency upon the above-described change quantity 
of pixel value (difference between pixel values of the same pixel), the direction of movement of the foreground is not 
taken into consideration, so whether or not correct soft key corresponding to that movement can be obtained was not 

35 clear. 

[0015] On the other hand, in the method disclosed in the Japanese Patent Application Laid Open No. 236347/1 993, 
an approach is employed to first generate hard key to shave, at respective points on the boundary between the fore- 
ground and the background, the side surface of that hard key (side surfaces of column in which the portion of a = 1 is 
caused to be the bottom surface shown in FIG. 1 A) by the elliptical cone to thereby generate soft key. In this case, the 

40 long axis and the short axis of the ellipse constituting the bottom surface of the elliptical cone is determined by motion 
vector of the remarked portion and the magnitude of the edge intensity. However, in this method, the direction of the 
edge intensity is not correctly taken into consideration. For this reason, whether or not precise or correct soft key 
corresponding to movement of the foreground an be obtained was not clear similarly to the above. 
[0016] Further, there is known from JP-A-4070293 a method for calculating the key signal in accordance with the 

45 relational color change speed, wherein the back color and a subject are discriminated from each other. 



Disclosure of the Invention 

[0017] It is an object of the present invention to generate a key signal which is precisely coping with motion blur, and 
so permits the influence of alias to be reduced. 

[0018] This object is achieved by a key signal generating apparatus, a picture synthesis apparatus, a key signal 
generating method and a picture synthesis method according to the enclosed independent claims. Advantageous 
features of the present invention are defined In the corresponding subclaims. 
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Brief Description of the Drawings 
[0019] 

5 FIG. 1 A is a view for expiaining hard key, and FIG. 18 is a view for explaining soft key. 

FIGS. 2A, 2B and 2C are views for explaining alias. 
FIGS. 3A and 3B are views for explaining motion blur. 

FIG. 4 is a view showing the state where local texture of picture is considered to be uniform. 
FIG. 5 is a view for explaining a method of calculating picture in which motion blur takes place. 
10 FIG. 6 is a view showing filtering (filter processing) for providing the effect of motion blur. 

FIG. 7 is a view showing foreground F and key signal a in which the effect of motion blur is taken into consideration. 
FIG. 8 is a view showing that local texture of the foreground F to which the effect of motion blur has been applied 
is considered to be uniform. 

FIG. 9 is a view showing that alias takes place in the case where pixel has finite size (dimensions). 
15 FIG. 1 0 is a block diagram showing the configuration of a first embodiment of a key signal generating apparatus 

to which this invention is applied. 

FIG. 11 is a flowchart for explaining the operation of the key signal generating apparatus of FIG. 10. 
FIG. 1 2 is a block diagram showing an example of the configuration of change detecting unit 1 0 of FIG. 1 0. 
FIG. 13 is a flowchart for explaining the operation of the change detecting section 10 of FIG. 12. 
20 FIG. 1 4 is a view showing sobel operator. 

FIG. 15 is a view for explaining processing of step S1-2 of FIG. 13. 

FIG. 1 6 is a block diagram showing an example of the configuration of feature point detecting section 20 of FIG. 1 0. 

FIG. 17 is a view for explaining processing of the feature point detecting section 20 of FIG. 16. 

FIG. 1 8 is a flowchart for explaining the operation of the feature point detecting section of FIG. 1 6. 
25 FIG. 1 9 is a block diagram showing an example of F-B detecting section 30 of FIG. 1 0. 

FIG. 20 is a view for explaining processing of very small area detecting section 31 of FIG. 1 9. 

FIG. 21 is a view for explaining processing of difference calculating section 34 of FIG. 19. 

FIG. 22 is a flowchart for explaining the operation of the F-B detecting section 30 of FIG. 19. 

FIG. 23 is a view for explaining processing of step S5 of FIG. 11 . 
30 FIG. 24 is a block diagram showing an example of the configuration of integral value calculating section 40 of FIG. 

10. 

FIG. 25 is a view for explaining processing of integral value calculating section 40 of FIG. 24. 
FIG. 26 is a flowchart for explaining the operation of the integral value calculating section 40 of FIG. 24. 
FIG. 27 is a block diagram showing an example of the configuration of key signal generating section 50 of FIG. 10. 
35 FIG. 28 is a view for explaining processing of the key signal generating section 50 of FIG. 27. 

FIG. 29 is a flowchart for explaining the operation of the key signal generating section 50 of FIG, 27. 
FIG. 30 is a block diagram showing the configuration of a second embodiment of the key signal generating appa- 
ratus to which this invention is applied. 

FIG. 31 is a block diagram showing an example of the configuration of feature point detecting section 60 of FIG. 30. 
40 FIG. 32 is a view for explaining processing of line thinning section 61 of FIG. 31 . 

FIG. 33 is a block diagram showing a third embodiment of the key signal generating apparatus to which this in- 
vention is applied. 

FIG. 34 is a view for explaining processing of integral value calculating section 70 of FIG. 33. 
FIG. 35 is a flowchart for explaining the operation of the key signal generating apparatus of FIG. 33. 
45 FIG. 36 is a block diagram showing an example of the configuration of the integral value calculating section 70 of 

FIG. 33. 

FIG. 37 is a flowchart for explaining the operation of integral value calculating section 70 of FIG. 36. 

FIG. 38 is a block diagram showing the configuration of a first embodiment of a picture synthesis apparatus to 

which this invention is applied. 

so FIG. 39 is a block diagram showing the configuration of a second embodiment of the picture synthesis apparatus 

to which this invention is applied. 

FIG. 40 is a block diagram showing an example of the configuration of information generating unit 90 of FIG. 39. 
Best Mode for Carrying Out the Invention 

55 

[0020] Prior to description of preferred embodiments of this invention, in order to clarify the correspondence rela- 
tionship between respective means of the invention described in the claims and the embodiments described below, 
the features of this invention will be described in the following manner in the state where the description of corresponding 
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embodiments (which should be considered to be as examples) are added within parentheses after respective means. 
[0021] Namely, a key signal generating apparatus according to this invention is directed to a key signal generating 
apparatus adapted for generating a key signal from picture, the apparatus comprising: change detecting means (e.g., 
change detecting section 10, shown in FIG. 10, 30 or 33, etc.) for detecting change of pixel value between pixels 

5 constituting picture; and key signal generating means (e.g., integral value calculating section 40 shown in FIG. 10 or 
30, or integral value calculating section 70 and key signal generating section 50 shown in FIG. 33, etc.). 
[0022] The key signal generating apparatus according to this Invention further comprises difference detecting means 
(e.g., F-B detecting section 30 shown In FIG. 10 or 30, etc.) for detecting difference between foreground picture and 
background picture constituting the picture to output difference information indicating the difference, wherein the change 

10 detecting means outputs change information Indicating change of pixel value, and the key signal generating means 
generates the key signal on the basis of the difference information and the change information. 
[0023] In the key signal generating apparatus according to this invention, the key signal generating means comprises 
normalizing means (e.g., normalization executing section 42 shown in FIG. 24, etc.) for normalizing the change infor- 
mation by the difference information to output normalized information obtained as the result of the normalization, and 

is integration executing means (e.g., integration executing section 43 shown in FIG. 24, etc.) for integrating the normalized 
information along a predetermined integral path). 

[0024] In the key signal generating apparatus according to this invention, the key signal generating means comprises 
range detecting means (e.g., integral path determining section 71 shown in FiG. 36, etc.) for detecting integral range, 
definite integral information generating means (e.g., definite integral value calculating section 72 shown in FIG. 36, 

20 etc.) for integrating the change information along a predetermined integral path within the integral range to generate 
definite integral information of the change information within the integral range, normalizing means (e.g., normalization 
executing section 73 shown in FIG. 36, etc.) for normalizing the change information by the definite integral information 
to output normalized information obtained as the result of the normalization, and integration executing means (e.g., 
integration executing section 74 shown in FIG. 36, etc.) for integrating the normalized information to thereby generate 

25 a key signal. 

[0025] A picture synthesis apparatus according to this invention is directed to a picture synthesis apparatus adapted 
for receiving a first picture consisting of at least foreground picture and background picture, and a second picture 
consisting of at least background picture to synthesize the foreground picture of the first picture and the background 
picture of the second picture, the apparatus comprising: change detecting means (e.g., change detecting section 10 

30 shown in FIG. 1 0, 30 or 33, etc.) for detecting change of pixel value between pixels constituting the first picture; key 
signal generating means (e.g., integral value calculating section 40 shown in FIG. 10 or 30, or integral value calculating 
section 70 shown in FIG. 33, etc.) for generating a key signal on the basis of the change of the pixel value outputted 
from the change detecting means; and synthesis means (e.g., mixer 85 shown in FIG. 38 or mixer 96 shown in FIG. 
39, etc.) for synthesizing the foreground picture of the first picture and the background picture of the second picture 

35 by using the key signal generated by the key signal generating means. 

[0026] The picture synthesis apparatus according to this invention further comprises difference detecting means (e. 
g., F-B detecting section 30 shown in FIG. 10 or 30, etc.) for detecting difference between the foreground picture and 
the background picture of the first picture to output difference information indicating the difference thus detected, where- 
in the change detecting means outputs change information indicating the change of the pixel value, and the key signal 

40 generating means generates the key signal on the basis of the difference information and the change information. 
[0027] It should be noted that it is a matter of course that the foregoing description does not mean that the respective 
means are limited to the above-mentioned components. 

[0028] The theoretical ground (principle) of this invention will now be described. It is to be noted that picture is as- 
sumed to be one-dimensional picture here for the brevity of explanation. 

45 [0029] It is now assumed that pixel value of pixel at position (coordinate) x constituting one-dimensional picture C. 
Moreover, the picture C is assumed to consist of foreground (foreground picture) F and background (background pic- 
ture) B, and pixel values of pixel at the position x constituting the foreground F and the background B are respectively 
designated at F(x) and B(x). It is further assumed that value at the position x of the key signal for extracting the fore- 
ground F from the picture C (contribution ratio of the foreground F at the position x) is a(x). In this case, it is assumed 

so that a(x) takes a value within the range from 0 to 1 , and respectively takes 1 and 0 at positions of pixels completely 
constituting the foreground F and the background B. In addition, at the positions of pixels where the foreground F and 
the background B are mixed, cc(x) is assumed to take a value corresponding to the degree of contribution of pixel value 
of the foreground F. 

[0030] In this case, the pixel value C(x) of pixels constituting the picture C is considered to be value in which the 
55 pixel value F(x) of pixels constituting the foreground F and the pixel value B(x) of pixels constituting the background 
B are synthesized at a ratio corresponding to the key signal a(x). Accordingly, the pixel value C(x) can be expressed 
by the following equation (1): 
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C(x) = a(x)F(x) + (1-a(x))B(x) (1) 

[0031] Here, the previously described alias takes place because pixel has definite size (dimensions), and motion 
5 blur takes place in the case where any one of the foreground F and the background B is moving, and in the case where 
both are independently moving. Explanation will now be given in a manner classified into four cases: 

[1 ] Case where the foreground F and the background B are stationary, and pixels can be considered to be infinitely 
small. 

10 [2] Case where the foreground F or the background B is moving, and pixels can be considered to be infinitely small. 

[3] Case where the foreground F and the background B are stationary, and pixels have definite size (dimensions). 
[4] Case where the foreground F or the background B is moving, and pixels have definite size (dimensions). 

[1] Case where the foreground F and the background B are stationary, and pixels are considered to be infinitely small. 

15 

[0032] In this case, since the foreground F and the background Bare stationary, no motion blurtakes place. Moreover, 
since pixels are considered to be infinitely small, no alias also takes place. Accordingly, the key signal a(x) takes any 
one of value of 0 and 1 , and the position x where that value becomes equal to 1 is in correspondence with the existing 
range of the foreground F. 

20 [0033] Here, when the equation (1) is differentiated by x, the equation (2) is obtained. It should be noted that de- 
scription of argument (x) of the function is omitted as the occasion may demand. 

C = (F-B)oc' + (F'-B')a + B' (2) 

25 

In the above equation, 1 (prime) represents differentiation. 

[0034] In the case where ordinary picture is assumed, texture (pixel value) of the foreground F and the background 
B is not uniform, but varies as shown in FIG. 4. However, in the case where the foreground F and the background B 
are locally viewed, e.g., in pixel units, etc., that texture can be considered to be uniform as shown in the FIG. 4 mentioned 
30 above. Accordingly, from a local point of view, the pixel value of the foreground F and the pixel value of the background 
B can be considered to be constant. As a result, the differential value P and the differential value B' can be both 
considered to be zero (0). From this fact, the equation (2) can be rewritten into the following equation (3): 



35 C'o(F-B)a' (3) 

[0035] Since the differential value C of the left side of the equation (3) is a value obtained by differentiating pixel 
value C(x) constituting the picture C by x, this differential value C represents change of pixel value between pixels 
constituting the picture C. It is seen from the equation (3) that change C'(x) of pixel value between pixels of the picture 
40 c and differential value a'(x) of the key signal have the proportional relationship in which difference F(x)-B(x) between 
pixel value F(x) of the foreground F and pixel value B(x) of the background B Is caused to be proportional constant. 

[2] Case where the foreground F or the background B is moving and pixels can be considered to be infinitely small. 

45 [0036] In the case where attention is now drawn to picture of moving object, motion blur takes place in the picture 
of the current frame as has been explained with reference to FIG. 3. Namely, the picture of the current frame is the 
picture in which movement of the object from the frame preceding (earlier) by one relative to the current frame up to 
the current frame is reflected. Accordingly, in the case where times of the current frame and the frame preceding by 
one are respectively designated at t^ t 1f and pixel value at time t of pixel at a certain position x of the picture C is 

so represented by Ct(x), pixel value Ct 2 (x) of pixel of the position x at the time ^ can be represented by time integration 
of picture Sx(t) which has been passed through the pixel of the position x for a time period from the times to t 2 . 
Namely, the pixel value Ctg(x) can be expressed by the following equation (4): 



55 



Ct a (x) n jsx{t)dt 

€1 



(4) 
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[0037] However, there is no way to find out (recognize) the picture Sx(t) which has been passed through the pixel 
of the position x for the time period from times tj to from the picture of the current frame, or the picture of the frame 
preceding by one. In view of the above, when the picture located at the position x is represented by St^x) at the time 
t 2 of the current frame, pictures existing within the range from the positions x., to x 2 are respectively represented by 

5 pictures St^) to St^Xg). In this case, when it is assumed that the pictures St 2 {^) to St 2 (x 2 ) have been passed through 
the pixel located at the position x for the time period from the times ^ to t^ whereby pixel value Ct 2 (x) of picture of the 
position x at the time t 2 has experienced the influence of the pictures St^x*,) to St 2 (x 2 ), the pixel value Ct^x) is equal 
to time integration of the picture Sx(t) which has been passed through the pixel of the position x for the time period 
from the times tj to t 2 as shown in FIG. 5B, and can be represented by the spatial integration of the pictures St^) to 

10 St2(x 2 ) existing within the range from the positions to x 2 at the time t 2 . Namely, the pixel value Ct 2 (x) can be expressed 
by the following equation (5): 



15 Ct,(x) = fst 2 (x)dx (5) 

*t 

[0038] Meanwhile, when still picture (picture in which no motion blur takes place) at the time t 2 is represented by S 
20 (x) and filter for giving the effect of the motion blur with respect to the still picture is represented by R(x), the right side 
of the equation (5) is equal to convolution integral of S(x) and R(x) as shown in FIG. 6. Accordingly, the equation (5) 
can be rewritten as indicated by the following equation (6). 

25 Ct 2 (x) = S(x)*R(x) (6) 

In the above equation, * indicates convolution integral. 

[0039] Now, filters which give the effect of the motion blur to the foreground F and the background B are assumed 
to be respectively represented by Rf(x) and Rb(x). In addition, in this embodiment, since the key signal a(x) moves 
30 along with the foreground F, in the case where the effect of the motion blur is given by the filter Rf(x) with respect to 
the foreground F, it is necessary to give the effect of the motion blur also to the key signal a(x) blur by the same filter 
Rf(x) as shown in FIG. 7B. When the effect of the motion blur is given to the foreground F, the background B and the 
key signal a in consideration of the above-mentioned facts to rewrite the equation (1), the following equation (7) is 
provided: 

35 

C = (a*Rf)(F*Rf) + (1-a*Rf)(B*Rb) (7) 
Differentiation of the above equation (7) gives the following equation (8): 

40 

C = (F*Rf - B*Rb)(a'*Rf) 
+ (F'*Rf - B'*Rb)(a*Rf) + B'*Rb (8) 

45 

[0040] If the texture of the picture of the foreground F to which the effect of the motion blur has been given, i.e., the 
texture of the picture obtained by filtering the foreground F by the filter Rf can be considered to be also uniform similarly 
to the case shown in FIG. 4 from a local point view. This similarly applies to the pixel value B of the background B. 
[0041] Accordingly, from a local point of view, the pixel values F*Rf and B*Rb which have been caused to undergo 
so filtering can be considered to be constant. As a result, these differential values (F*Rf)' and (B*Rb)' can be both con- 
sidered to be zero (0), From this fact, the above-mentioned equation (8) can be rewritten as indicated by the following 
equation (9): 

55 C » (PRf-B*Rb)(a*Rf)' (9) 

[0042] It is seen from the equation (9) that change C of the pixel value between pixels of the picture C and differential 
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value (a *Rf)' of the key signal in which the effect of the motion blur is taken into consideration have the proportional 
relationship in which difference F*Rf-B*Rb between F*Rf of the foreground and the pixel value B*Rb of the background 
in which the effect of the motion blur is similarly taken into consideration is caused to be proportional constant. 

s [3] Case where the foreground F and the back ground B are stationary, and pixels have definite size 

[0043] In the case where the size of pixel is considered to be Infinitely small, the pixel value smoothly varies with 
respect to the position x as shown in FIG. 9A. In contrast, in the case where pixels have definite size, the pixel value 
varies in a step form with respect to the position x as shown in FIG. 9B. For this reason, in the case where the signai 

10 shown in FIG. 9A is merely sampled, alias takes place at the boundary between the foreground F and the background 
B as previously described. In order to reduce such alias, there is a method, as shown in FIG. 9A, in which, for the 
purpose of calculating the pixel value, the pixel value is caused to undergo filtering by the low-pass filter G(x) to carry 
out sampling, i.e., value obtained as the result of convolution integral of the pixel value and the low-pass filter G(x) at 
the pixel position is caused to be new pixel value. In view of the above, in the case where picture in which alias is 

15 reduced by the low-pass filter G(x) in a manner described above is assumed, its pixel value (new pixel value) C(x) can 
be represented by the following equation (10) from the above-mentioned equation (1): 



20 



25 



C(x) = (a(x)F(x) + (1 -a(x))B(x))*G(x) (1 0) 

[0044] When the equation (10) is differentiated by x, the following equation (11) is provided. 

C = ((F-B)a' +- (F'-B'Ja+BTG 
= ((F-B)a')*G + ((F*G)'-(B*G) , )a 

+ (B*G)' (11) 

30 [0045] Here, the texture of the picture obtained as the result of the fact that the pixel value of the foreground F and 
the pixel value of the background B are subjected to filtering by the low-pass filter R can be also considered to be both 
uniform similarly to the case shown in FIG. 4 when they are locally viewed. 

[0046] Accordingly, from a local point of view, the pixel values F*G and B*G which have been caused to undergo 
filtering can be considered to be both constant. As a result, these differential values (F*G)' and (B*G)' can be considered 
35 to be both zero (0). From this fact, the equation (11) can be rewritten as indicated by the following equation (12). 

C = ((F-B)a')*G (12) 

40 [0047] Moreover, when locally viewed, as explained with reference to FIG. 4, the pixel values of the foreground F 
and the background B can be considered to be both constant. Accordingly, the equation (12) can be rewritten as 
indicated by the following equation (13): 



45 



C = (F-B)(a*G)' (13) 



[0048] It is seen from the above equation (1 3) that change C of the pixel value between pixels of the picture C and 
the differential value (oc*G)' of the key signal for reducing alias have the proportional relationship in which difference 
F-B between the pixel value of the foreground F and the pixel value of the background B is caused to be proportional 
so constant. 

[4] Case where the foreground F or the background B is moving and pixels have definite size 

[0049] In this case, from the above-described equations (1), (7) and (10), the pixel value C can be expressed by the 
55 following equation (14): 

C = ((<x*Rf)(F*Rf) + (1-a*Rf)(B*Rf))*G (14) 
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[0050] When the equation (14) is differentiated by x, the following equation (15) is provided: 

C = ((F*Rf-B*Rb)(a'*Rf) 

5 + (F'*Rf-B'*Rb)(a*Rf)+B , *Rb)*G (15) 

[0051] Here, the texture of the picture obtained by filtering the pixel value of the foreground F by the filter Rf to further 
allow it to undergo filtering by the low-pass filter G can be considered to be also uniform similarly to the case in FIG. 

10 4 when locally viewed. This similarly applies to the pixel value of the background B. 

[0052] Accordingly, from a local point of view, the pixel values F*Rf*G and B*Rb*G which have been caused to 
undergo filtering can be considered to be constant. As a result, these differential values (F*Rf*G)', (B*Rb*G)' can be 
considered to be both zero (0). Further, when locally viewed, as explained with reference to FIG. 4, the pixel value of 
the foreground F and the pixel value of the background B can be considered to be both constant. From this fact, the 

15 equation (15) can be rewritten as indicated by the following equation (1 6): 

C = (F*Rf - B*Rb)(a*Rf*G)' (1 6) 

20 [0053] It is seen from the above equation (1 6) that the change C of the pixel value between pixels of the picture C 
and the differential value (a*Rf*G)' of the key signal such that the effect of motion blur is taken into consideration and 
that alias is permitted to be reduced have proportional relationship in which difference F*Rf-B*Rb between the pixel 
value F*Rf and the pixel value B*Rb of the background in which the effect of motion blur is taken into consideration is 
caused to be the proportional constant. 

25 [0054] It is seen from the above-mentioned equations (3), (9), (1 3) and (1 6) that, in all of the above-described cases 
of [1] to [4], the change C of the pixel value between pixels of the picture C and the differential value a' of the key 
signal have the proportional relationship, and the proportional constant is difference F-B between the pixel value t of 
the foreground and the pixel value of the background constituting the picture C. Namely, even if motion blur or alias 
does not take place in the picture C, the proportional relationship between C and a* holds. Moreover, in the case where 

30 motion blur or alias takes place in the picture C, the proportional relationship between C and a' holds also with respect 
to the relationship between the key signal a in which the influence of that motion blur is taken into consideration, or 
the key signal for permitting alias to be reduced and the pixel value C. In any case, the proportional constant becomes 
difference F-B between the pixel value of the foreground and the pixel value of the background constituting the picture C. 
[0055] Accordingly, a key signal a adapted to allow the proportional relationship to hold is such that in the case where 

35 motion blur takes place In the picture C, it becomes a key signal in which the influence of that motion blur is taken into 
consideration, and that in the case where alias takes place in the picture C, it becomes a key signal for allowing that 
alias to be reduced. This invention is based on the above-described theoretical ground (principle). 
[0056] FIG, 1 0 shows the configuration of an embodiment of the key signal generating apparatus adapted for gen- 
erating a key signal on the basis of the above-mentioned principle. A picture C for generating key signal is inputted to 

40 a change detecting section 10. Then, the change detecting section 10 calculates (detects) change C (change infor- 
mation) of pixel value between pixels of the picture C to deliver this change C to a feature point detecting section 20 
and an integral value calculating section 40. 

[0057] The feature point detecting section 20 detects closed curve for path determination and feature point which 
will be described later on the basis of the change C of the pixel value delivered from the change detecting section 1 0 
45 to deliver the path determination closed curve and the feature point to an F-B detecting section 30 and an integral 
value calculating section 40. 

[0058] The F-B detecting section 30 is supplied with the change C of the pixel value from the change detecting 
section 10, and the path determination closed curve and the feature point from the feature point detecting section 20, 
and is supplied with the same picture C as the picture inputted to the change detecting section 10. Thus, the F-B 
so detecting section 30 calculates (detects) difference F-B (difference information) between the pixel value of the fore- 
ground F and the pixel value of the background B of the picture C to deliver this difference F-B to the integral value 
calculating section 40. 

[0059] The integral value calculating section 40 calculates the skeleton portion which will be described later of key 
signals a corresponding to the entirety of the picture C on the basis of the change C of the pixel value delivered from 
55 the change detecting section 1 0, the path determination closed curve and the feature point delivered from the feature 
point detecting section 20, and the difference F-B delivered from the F-B detecting section 30 to deliver this skeleton 
portion to a key signal generating section 50. 

[0060] The key signal generating section 50 extracts the foreground F from the picture C by using the skeleton portion 



9 



EP0 771 107 B1 



delivered from the integral value calculating section 40, and to generate a key signal for synthesizing the extracted 
foreground F with any other picture. Namely, the key signal that the integral value calculating section 40 outputs is only 
the skeleton portion of the key signals a corresponding to the entirety of the picture C. The key signal generating 
section 50 interpolates portions except for the skeleton portion of the key signal to generate the key signal a corre- 
5 sponding to the entirety of the picture C. 

[0061] More practical operation of the key signal generating apparatus will now be described with reference to the 
flowchart of FIG. 11 . It is to be noted that picture C is assumed to be picture on the xy plane, i.e., two-dimensional 
picture in the following description. 

[0062] At step S1 , when picture C for generating a key signal is inputted, that picture C is delivered to the change 
10 detecting section 1 0 and the F-B detecting section 30. Then, the change detecting section 1 0 is operative so that when 
picture C is inputted thereto, it calculates change C of the pixel value between pixels of that picture C. Namely, at this 
step S1, with respect to respective pixels constituting the picture C, e.g., distance on the color space between pixels 
adjacent to each other is calculated. The distance thus calculated is outputted as change (color change in this case) 
C of the pixel value between pixels of the picture C. 
is [0063] A more practical example of the configuration of the change detecting section 1 0 of FIG. 1 0 is shown in FIG. 
12. The picture C is delivered to an RGB decomposing section 11. The RGB decomposing section 11 decomposes 
the picture C into, e.g., components of respective axes directions in RGB (Red, Green, Blue) space, i.e., R-component 
G-component and B-component (hereinafter referred to as R, G, B components as the occasion may demand). These 
R, G, B components are respectively delivered to first (1 -st) order differential value calculating sections 1 2R, 1 2G and 

20 12B. 

[0064] The first order differential value calculating sections 1 2R, 1 2G, 1 2B calculates respective first order differential 
values R\ G\ B' to calculate directions and magnitudes (vectors) R\ G', B' of changes of respective components (i.e., 
gradients of the curve assumed to be constituted by pixel values (respective components) of the picture C on the space 
in which axis indicating pixel values vertical to the xy-plane where the picture C exists is added to the xy-plane. Then, 
\ 25 vectors R', G\ B' of respective changes of the R, G, B components are delivered to a vector synthesis section 1 3. 

[0065] The vector synthesis section 13 synthesizes the vectors R\ G\ B' delivered from the first order differential 
value calculating sections 12R, 12G, 12B, i.e., synthesizes directions of respective changes of R, G, B components, 
and synthesizes magnitudes of the respective components to calculate change (color change) C of pixel values be- 
tween pixels of the picture C. 

30 [0066] The operation of the change detecting section 1 0 will now be described with reference to the flowchart of FIG. 
13. 

[0067] At step S1-1, the RGB decomposing section 11 decomposes the picture C into components of respective 
axes directions of R G B space, i.e., R, G, B components. It is to be noted that the picture C may be decomposed into 
Y, U, V components in color space except for the RGB space, e.g., YUV space. Namely, the color space employed is 

35 not particularly limited. In addition to decomposing the picture C into components of respective axes directions of the 
color space, luminance components may be extracted therefrom to carry out processing of the succeeding stage only 
with respect to the luminance components. Then, the R, G, B components obtained at the step S1-1 are respectively 
delivered from the RGB decomposing section 11 to the first order differential value calculating sections 12R, 12G, 12B. 
[0068] At step S1-2, the first order differential value calculating sections 12R, 12G, 12B is operative so that when 

40 the R, G, B components are delivered thereto, they calculate the first order differential values R*. G\ B' of these R, G, 
B components. 

[0069] It is to be noted that, in this embodiment, first order differential values of respective components are determined 
by carrying out, e.g., convolution of respective components and sobel operator, etc. Namely, at the first order differential 
value calculating sections 12R, 12G, 12B, values obtained by carrying out convolution of x or y direction components 
45 of the respective R, G, B components and sobel operator SO x or SO y shown in FIG. 14 are caused to be respectively 
first order differential values f x , f y In the x or y direction. 

[0070] First order differential values of the R, G, B components determined in a manner described above, i.e., chang- 
es of the R, G, B components are such that their magnitudes (gradients) are maximum at respective pixels. Accordingly, 
the directions of changes of respective components are in correspondence with the directions in which the edge in- 
50 tensity is maximum. Namely, at this step S1-2, as shown in FIG. 15, changes of the R, G, B components with respect 
to the direction caused to be in correspondence with the direction in which the edge intensity is maximum of any 
(arbitrary) directions of 360 degrees with respect to the remarked pixel (the portion to which slanting lines are attached 
in FIG. 15) are determined. 

[0071] Then, first order differential values R', G\ B" of R, G, B components respectively determined at the first order 
55 differential value calculating sections 12R, 12G, 12B (these values are vectors because they have magnitude and 
direction as described above) are delivered to the vector synthesis section 13. 

[0072] At step S1 -3, the vector synthesis section 13 synthesizes first order differential values R\ G', B' to calculate 
change (color change) C of the pixel value between pixels of the picture C. Namely, if first order differential values R\ 
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G\ B' are respectively represented by (r x , r y ), (g x , g y ), (b x , b y ), and C is represented by (c x , Cy) (a and b of (a, b) 
respectively indicate x-component and y-component). At this step S1 -3, x component c x and y component Cy of change 
C of the pixel value are respectively determined by the expression r x +g x +b x and ryi-gy+by. Then , change C of the pixel 
value determined in a manner as stated above at the change detecting section 10 is delivered to the feature point 
5 detecting section 20 and the integral value calculating section 40. Thus, the processing at the change detecting section 
1 0 is completed. 

[0073] At step S2 (FIG. 11), the feature point detecting section 20 is operative so that when the change C of the 
pixel value is delivered thereto from the change detecting section 10, it detects the path determination closed curve 
and the feature point from the change C of that pixel value. 

10 [0074] A more practical example of the configuration of the feature point detecting section 20 is shown in FIG. 1 6. 
A boundary (border) detecting section 21 is supplied with the change C of the pixel value from the change detecting 
section 1 0. Thus, the boundary detecting section 21 detects, from the change C of the pixel value, boundary between 
the foreground (object) and the background B which constitute the picture C. Namely, the boundary detecting section 
21 is operative, as shown in FIG. 1 7A, for example, to detect the points (pixel) where the change C of the pixel value 

15 is well to detect, as the boundary between the foreground F and the background B (boundary of the object in the picture 
C), closed curve constituted by connecting such points (since this closed curve is used for determining integral path 
which will be described later, it will be called hereinafter path determination closed curve as the occasion may demand). 
Then, the path determination closed curve detected at the boundary detecting section 21 is delivered to the feature 
point determining section 22. 

20 [0075] The feature point determining section 22 determines, as the feature point, predetermined points on the path 
determination closed curve delivered from the boundary detecting section 21 as shown in FIG. 17B, for example. 
Namely, the feature point determining section 22 detects, e.g., the points where curvature is high of points constituting 
the path determination closed curve to determine those points as the feature point 

[0076] Moreover, the feature point determining section 22 is operative, as shown in FIG. 1 7C, for example, to deter- 
25 mine the closed curve surrounding the path determination closed curve (the portion indicated by dotted lines in FIG. 
17C), and divides the area of the background B surrounded by that closed curve and the path determination closed 
curve into several areas where the change C* of that pixel value is relatively constant (hereinafter referred to as back- 
ground constant area as the occasion may demand). Further, the feature point determining section 22 partitions the 
path determination closed curve in the background constant area units so that divided very small areas are provided. 
30 In addition, the feature point determining section 22 detects middle points in respective very small areas to determine 
those points as the feature point. 

[0077] The operation of the feature point detecting section 20 will now be described with reference to the flowchart 
of FIG. 18. 

[0078] At step S2-1 , the boundary detecting section 21 is operative so that when the change C* of the pixel value is 
35 delivered thereto from the change detecting section 1 0, it detects the points where the change C of that pixel value is 
great to connect those points to thereby detect a path determination closed curve. Then, the path determination closed 
curve is delivered to the feature point determining section 22. 

[0079] At step S2-2, the feature point determining section 22 is operative so that when the path determination closed 
curve is delivered thereto from the boundary detecting section 21 , it determines, as the feature point, predetermined 
40 points on the path determination closed curve. Then, the path determination closed curve and the feature points re- 
spectively obtained at the boundary detecting section 21 and the feature point determining section 22 are delivered to 
the F-B detecting section 30 and the integral value calculating section 40. Thus, the processing at this feature point 
detecting section 20 is completed. 

[0080] At step S3 (FIG. 11), the F-B detecting section 30 is operative so that when the path determination closed 
45 curve and the feature points are delivered thereto from the feature point detecting section 20, it calculates difference 
F-B between the pixel value of the foreground F and the pixel value of the background B which constitute the picture C. 
[0081] A more practical example of the configuration of the F-B detecting section 30 is shown in FIG. 1 9. The very 
small area detecting section 31 is supplied, in addition to the picture C (picture value), with the feature points and the 
path determination closed curve from the feature point detecting section 20. Thus, the very small area detecting section 
so 31 detects the very small area including the feature points as shown in FIG. 20A, for example. Namely, the very small 
area detecting section 31 divides the path determination closed curve into very small line segments at points equi- 
distant from the feature point as shown in FIG. 20B in the state where a portion of FIG. 20A is enlarged to detect, as 
the very small area (the portion to which slanting lines are attached in FIG. 20B), the range spaced by a predetermined 
distance (e.g., distance corresponding to four pixels, etc.) toward the side of the foreground F or the background B 
55 from that line segment. Moreover, the very small area detecting section 31 extracts, with respective to respective very 
small areas, pixels constituting the foreground F included within those areas (pixels existing at the foreground F side 
from the path determination closed curve, i.e., the portion to which slanting lines going down to the left are attached 
in FIG. 20B) to deliver pixel value of those pixels (hereinafter referred to as F boundary pixel value as the occasion 
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may demand) to a color distribution forming circuit 32F ( and extracts pixels constituting the background B (pixels existing 
at the background B side from the path determination closed curve, i.e., the portion to which slanting lines going down 
to the right are attached in FIG. 20B) to deliver pixel value of those pixels (hereinafter referred to as B boundary pixel 
value as the occasion may demand) to a color distribution forming section 32B. 

5 [0082] The color distribution forming sections 32F, 32B respectively determines color distributions on the colorspace, 
e.g., RGB space, etc. of the F boundary pixel values and the B boundary pixel values every very small areas delivered 
from the very small area detecting section 31 to respectively deliver them to center of gravity detecting sections 33F, 
33B. The center of gravity detecting sections 33F, 33B respectively determine centers of gravity of the color distributions 
of the F boundary pixel values and the B boundary pixel values delivered from the color distribution forming sections 

10 32F, 32B to considers respective centers of gravity as pixel values of the foreground F and the background B of pixels 
in the gray area to deliver them to a difference calculating section 34. 

[0083] The difference calculating section 34 is operative, as shown in FIG. 21 , for example, to subtract, from the 
center of gravity of the color distribution of the F boundary pixel values delivered from the center of gravity detecting 
section 33F, the center of gravity of the color distribution of the B boundary pixel values delivered from the center of 
15 gravity detecting section 33B to output the subtraction result as difference F-B between the foreground F and the 
background B (different of pixel value between the pixel value of the foreground F and pixel value of the background 
B, and this difference is thus considered to be vector similarly to C) in the very small area (or at feature point included 
within that very small area). 

[0084] The operation of the F-B detecting section 30 will now be described with reference to the flowchart of FIG. 22. 
20 [0085] At step S3-1 , the very small area detecting section 31 detects the very small area, which has been explained 
with reference to FIG. 20, on the basis of the feature points and the path determination closed curve delivered from 
the feature point detecting section 20 to further extract F boundary pixel values and B boundary pixel values in that 
very small area to respectively deliver the F bou ndary pixel value and the B boundary pixel value to the color distribution 
forming sections 32F, 32B. 

25 [0086] At step S3-2, the color distribution forming sections 32F, 32B respectively determines color distributions on 
the RGB space of the F boundary pixel values and the B boundary pixel values in a certain very small area to deliver 
them to the center of gravity detecting sections 33F, 33B. 

[0087] At step S3-3, the center of gravity detecting sections 33F, 33B respectively calculates (detect) centers of 
gravity of color distributions of the F boundary pixel values and the B boundary pixel values to deliver them to the 

30 difference calculating section 34. 

[0088] At step S3-4, the difference calculating section 34 determines difference value between the center of gravity 
of the color distribution of the F boundary pixel values delivered from the center of gravity detecting section 33F and 
the center of gravity of the color distribution of the B boundary pixel values delivered from the center of gravity detecting 
section 33B to output this difference value as difference F-B between the foreground F and the background B at the 

35 feature point included within the very small area. 

[0089] At step S3-5, after the difference F-B between the foreground F and the background at the feature point 
included within a certain very small area is obtained in a manner as stated above, whether or not the difference F-B 
is determined with respect to all very small areas detected at the very small area detecting section 31 (all feature points 
detected at the feature point detecting section 20) is judged. In the case where ft is judged at the step S3-5 that the 

40 difference F-B has not yet been determined with respect to all very small areas, the processing operation returns to 
the step S3-2 to carry out the processing at step S3-2 and steps subsequent thereto with respect to the very small 
areas where the difference F-B has not been determined. On the other hand, in the case where it is judged at the step 
S3-5 that the difference has been determined with respect to all the very small areas, the processing at the F-B detecting 
section 30 is completed. Then, the differences F-B which have been determined with respect to all the very small areas 

45 are delivered to the integral value calculating section 40. 

[0090] It is to be noted that the direction of the difference F-B (vector) between the foreground F and the background 
B obtained in a manner as described above is in correspondence with (substantially in correspondence with) the di- 
rection in which the edge intensity becomes maximum, and the direction of F-B and the direction of C are in corre- 
spondence with each other (substantially in correspondence with each other). 

so [0091] Further, while the F-B detecting section 30 shown in FIG. 1 9 determines the very small area on the basis of 
the path determination closed curve (the boundary between the foreground F and the background B) obtained from 
the change C of the pixel value to determine the difference F-B from the very small area, the difference F-B may be 
determined, e.g., in a manner stated below at the F-B detecting section 30. In more practical sense, e.g., not only the 
frame of the picture C but also frames constituting moving picture before and after that frame is inputted to the F-B 

55 detecting section 30 in a time series manner. Then, the F-B detecting section 30 stores the frame inputted at present 
until the subsequent frame is inputted. In the case where picture C for generating a key signal is inputted, frame of 
that picture C and the stored frame preceding (earlier) by one are caused to undergo the so-called block matching to 
detect the boundary between the foreground F and the background B on the basis of the above-mentioned result and 
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the result of the block matching preceding thereto as occasion demands, Then, the F-B detecting section 30 determines 
feature points on the path determination closed curve estimated from the boundary by using the boundary to detect 
the very small area including respective feature points similarly to the case at the steps S3-1, S3-2 of FIG. 22 to 
recognize respective distributions of the foreground F and the background B which constitute the picture C every 

5 respective very small areas to determine centers of gravity of respective color distributions of the foreground F and 
the background B similarly to the case at the steps S3-3, S3-4 of FIG. 22 at times subsequent thereto to further deter- 
mine direction of the position of the center of gravity of the color distribution of the background F with respect to the 
position of the center of gravity of the color distribution of the background B, and distance on the color space therebe- 
tween, i.e., the difference F-B between the fore is unnecessary to deliver the path determination closed curve and the 

10 feature points from the feature point detecting section 20 to the F-B detecting section 30. It is to be noted that the detail 
of the method of detecting the boundary between the foreground F and the background B by using block matching as 
described above is disclosed, e.g., in the Japanese Patent Application No. 164792/1995, etc. 
[0092] Moreover, while the method of detecting the boundary between the foreground F and the background B is 
not limited to the method using change C of the pixel value and the method using block matching as described above, 

15 detection of the boundary between the foreground F and the background B may be carried out by instruction from 
users as described later in FIG. 30, for example. Further, the boundary between the foreground F and the background 
B may be detected by employing in combination, e.g., the method by instruction from users and the method using 
change C of pixel value. Furthermore, the boundary between the foreground F and the background B may be detected 
by employing in combination, e.g., the method by instruction from users and the method using block matching. Namely, 

20 e.g., an approach may be employed to detect the boundary between the foreground F and the background B by in- 
struction from users with respect to the first frame, and to detect the boundary between the foreground F and the 
background B by block matching with respect to frames subsequent thereto. 

[0093] Further, the method of detecting the difference F-B is not limited to the above-described method, but may 
employ other methods. 

25 [0094] Moreover, while after the processing of the step S2, the processing of the step S3 is carried out in the flowchart 
shown in FIG. 11 , in the case where there is no necessity of delivering the path determination closed curve and the 
feature points with respect to the F-B detecting section 30 from the feature point detecting section 20, the processing 
of the step S2 and the step S3 may be carried out in an order opposite to the above, or may be carried out at the same 
time (in parallel). The difference F-B determined in a manner as described above at the step S3 (FIG. 11) is delivered 

30 from the F-B detecting section 30 to the integral value calculating section 40. 

[0095] At step S4, the integral value calculating section 40 normalizes change C of pixel value delivered from the 
change detecting section 10 by absolute value (F-B! of difference F-B delivered from the F-B detecting section 30. 
[0096] At step S5, the integral value calculating section 40 integrates the normalized value C'/IF-BI along, e.g., the 
direction In which change C of pixel value is calculated (detected) to thereby calculate the skeleton portion of the key 



[0097] In more practical sense, since the change C of pixel value between pixels of the picture C and the differential 
value a 1 of the key signal have the proportional relationship therebetween as described above, and the proportional 
constant is difference F-B between the foreground and the background which constitute the picture C, i.e., the expres- 
sion C'=(F-B) a' holds, the key signal cc(p) can be determined by the following equation (17). 



[0098] In the above equation, oc(p) means a key signal with respect to pixel at the position (point) p(x, y) constituting 
45 the picture C, and F(p) and B(p) respectively means pixel values of pixels at the position p(x, y) of the foreground F 
and the background B which constitute the picture C. Further, C(p) means pixel value of pixel at the position p(x, y) 
constituting the picture C. It is to be noted that description of the argument (p) of the function will be hereinafter omitted 
as the occasion may demand. 

[0099] Accordingly, at the above-described step S4, C'/IF-BI constituting the right side of the equation (17), i.e., 
50 normalized value obtained by normalizing the change C of the pixel value by the absolute value [F-B! of difference is 
determined. At this step S5, integration at the right side of the equation (17), i.e., integration of the normalized value 
CYIF-BI along, e.g., the direction in which change C of the pixel value is detected is carried out. Value obtained as the 
result of integrating value obtained by normalizing change C of the pixel value by the absolute value IP-Si of difference 
without integrating the change C of the pixel value as it is, i.e. , the key signal a takes value within the range from 0 to 1 . 
55 [0100] The integration carried out at the step S5 will now be described in more detail. When space in which the z- 
axis perpendicular to the xy plane where the picture C exists is provided is assumed to be on the xy-plane is assumed 
as shown in FIG. 23, for example, to take pixel value in the z-axis direction, plane (curved surface) V consisting of pixel 
values is formed within that space. It should be noted that if the picture C is color picture (picture image), there are 



35 signal a. 



40 




C(p) 



(17) 
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three components such as RGB or YUV, etc. as described above, but the pixel value is assumed to consist of one 
component for the brevity of explanation. 

[0101] In this case, gradient at an intersecting point of a straight line passing through a certain point p on the picture 
C and perpendicular to the xy-plane and the plane V, i.e., change of pixel value C(p) of pixel located at the position p 
5 can be represented as vector in the maximum inclination direction when the plane V is viewed from that point C(p) 
(vector obtained by calculating gradient of the plane V at the point C(p)). 

[01 02] Here, the maximum inclination direction Is the direction In which in the case where a ball having size assumed 
to be infinitely small is placed at the point C(p), that ball roils. 

[0103] Let now suppose that in the case where a ball is placed at the point C(p) and that ball falls (rolls) down to the 
10 point C(p 0 ), the locus (path) that the ball depicts on the plane V is U. In this instance, the integration of the equation 
(17) means the integration along the path (hereinafter referred to as the maximum inclination direction path as the 
occasion may demand), and the equation (1 7) means that the integration expressed thereby is carried out, whereby 
key signal a(p) with respect to the pixel located at the position p can be determined. 

[0104] Accordingly, if the key signal a(p 0 ) with respect to the pixel located at the position p 0 can be recognized, it is 
15 possible to determine the key signal a(p) with respect to the pixel located at the position p in accordance with the 
following equation (18) from the equation (17): 



[0105] In the above equation, integral symbol in which U is attached at the upper right means the integration along 
the maximum inclination direction path U. 

[0106] At step S5, e.g., calculation of the right side of the equation (18) is carried out, or the like, whereby the key 
signal a (the skeleton portion of the key signal) is determined. 

25 [0107] It is to be noted that it is not necessarily required to carry out the integration of the right side of the equation 
(18) along the maximum inclination direction path U, Namely, integration along path (FIG. 23) which is not the 
maximum inclination direction path U as the path from the point C(p) up to the point C(p 0 ) may be carried out. Namely, 
since the integration of the right side of the equation (18) means the integration of gradients at respective points on 
the maximum inclination direction path U , in the case where integration is carried out along other path , the component 

30 j n the direction of the maximum inclination direction path U of gradients at respective points on the path U 1 is integrated 
(Accordingly, this integration can be also considered to be eventually the same as the integration along the maximum 
inclination direction path U), thereby making it possible to determine the same key signal a as that in the case where 
integration is carried out along the maximum inclination direction path U. 

[0108] It is to be noted that since in the case where integration is carried out along paths except for the maximum 
35 inclination direction path U, the quantity of operations is increased to more degree as compared to the case where 
integration is carried out along the maximum inclination direction path, it is preferable to carry out integration along the 
maximum inclination direction path U as far as possible. 

[0109] An example of the configuration of the integral value calculating section 40 of FIG. 10 for carrying out a 
processing as described above is shown in FIG. 24. The path determination closed curve and the feature points from 
40 the feature point detecting section 20 are delivered to an integral path determining section 41 , and change C of the 
pixel value from the change detecting section 10 and difference F-B from the F-B detecting section 30 are delivered 
to a normalization executing section 42. 

[0110] The integral path determining section 41 determines integral path for carrying out integration of the equation 
(1 8) on the basis of the path determination closed curve and the feature points delivered from the feature point detecting 

45 section 20. Namely, as shown in FIG. 25A, the integral path determining section 41 determines, as the integral path 
with respect to corresponding feature point, a line in a normal direction of the path determination closed curve and 
passing through the corresponding feature point. The integral path with respect to the feature point determined in a 
manner as described above is delivered to the normalization executing section 42 and the integration executing section 
43. It is to be noted that the integral path determined in this way is the path substantially along the maximum inclination 

50 direction path U. 

[0111] The normalization executing section 42 determines normalized value C'/IF-BI (normalized information) ob- 
tained by normalizing change C of pixel value of pixel existing on the integral path delivered from the integral path 
determining section 41 by absolute value IF-BI of difference F-B to deliver it to the integration executing section 43. 
The integration executing section 43 carries out integration of the normalized value C'/IF-BI along the integral path 
55 from the integral path determining section 41 in accordance with the equation (18) to determine key signal a on the 
integral path. The key signal on the integral path is delivered to the closed curve generating section 44. 
[0112] The closed curve generating section 44 generates a P1 closed curve and a P2 closed curve which will be 
described later from the key signal a delivered from the integration executing section 43. 
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[01 1 3) Namely, while the Integration in accordance with the equation (1 8) is carried out along the integral path shown 
in FIG. 25A at the integration executing section 43, whereby the key signal a on the integral path can be obtained, that 
key signal a becomes equal to 0 (zero) at the boundary between the gray area and the background B, and becomes 
equal to 1 at the boundary between the gray area and the foreground F. Accordingly, as shown in FIG. 25A, the closed 

5 curve generating section 44 determines points where key signals a on respective integral paths become equal to 0 
(points where a is equal to 0, which will be referred to as p 1 as the occasion may demand hereinafter), and determines 
points where key signals a become equal to 1 (points where a becomes equal to 1 , which will be referred to as p 2 as 
the occasion may demand hereinafter), thus to connect the points p, on respective integral paths to each other, and 
to connect the points p 2 on the respective integral path to each other Then, as shown in FIG. 25B, the closed curve 

10 generating section 44 outputs, as P1 closed curve, the closed curve obtained by connecting the points p 1 to each other, 
and outputs, as P2 closed curve, the closed curve obtained by connecting the points p 2 to each other. The key signal 
a obtained at the integration executing section 43 and the P1 closed curve and the P2 closed curve obtained at the 
closed curve generating section 44 are delivered to the key signai generating section 50 (FIG. 10). 
[01 1 4] The operation of the integral value calculating section 40 will now be described with reference to the flowchart 

15 of FIG. 26. 

[0115] At step S4-1, the integral path determining section 41 allows any one of feature points delivered from the 
feature point detecting section 20 to be remarked feature point to determine a straight line passing through the remarked 
feature point and in normal direction of the path determination closed curve delivered from feature point detecting 
section 20 as the integral path with respect to the remarked feature point. The integral path with respect to the remarked 

20 feature point is delivered to the normalization executing section 42 and the integration executing section 43. 

[0116] At step S4-2, the normalization executing section 42 divides the change C of the pixel value of the pixel 
existing on the integral path delivered from the integral path determining section 41 by absolute value IP-Si of the 
difference which is the difference F-B delivered from the F-B detecting section 30 and is the difference determined 
from the very small area (FIG. 20) including the remarked feature point to deliver the normalized value CVIF-BI obtained 

25 as the result thereof to the integration executing section 43. 

[0117] At step S4-3, the integration executing section 43 integrates the normalized value C'/.F-b! in accordance with 
the equation (18) along the integral path from the integral path determining section 41 by using the integral path and 
the normalized value CVIF-BI with respect to the pixel existing on the integral path respectively delivered from the 
integral path determining path and the normalization executing section 42 to calculate a key signal a on that integral 

30 path. This key signal a is delivered to the closed curve generating section 44. In this case, at the integration executing 
section 43, e.g., the remarked feature point is regarded as the point where a becomes equal to 0.5, and the integration 
of the equation (1 8) is carried out with the remarked feature point being as the initial point. Accordingly, in this example, 
the integration of the equation (1 8) is carried out in the state where ct(p 0 ) is caused to be 0.5. 
[0118] At step S4-4, the integral path determining section 41 judges whether or not all of feature points delivered 

35 from the feature point detecting section 20 have been caused to be the remarked feature point. In the case where the 
integral path determining section 41 judges that all of the feature points from the feature point detecting section 20 
have not yet been caused to be the remarked feature point, its processing operation returns to the step S4-1 . As a 
result, at the integral path determining section 41 , feature point which is not yet caused to be the remarked feature 
point is newly caused to be the remarked feature point. At times subsequent thereto, processing as described above 

40 will be repeated. 

[0119] On the other hand, in the case where the integral path determining section 41 has judged at the step S4-4 
that all the feature points have been already to be caused to be the remarked feature point, i.e., in the case where the 
integration of the equation (18) along all of Integral paths passing through respective feature points delivered from the 
feature point detecting section 20 is carried out, whereby key signals a on those integral paths (hereinafter referred to 

45 as the skeleton portion as the occasion may demand), the processing proceeds to step S4-5. 

[01 20] At the step S4-5, the closed curve generating section 44 detects the point p 1 (the point of a = 0) and the point 
p 2 (the point of a = 1) from the key signals a on respective integral paths delivered from the integration executing 
section 43 to connect the points p^ to each other and to connect the point p 2 to each other to thereby generate a P1 
closed curve and a P2 closed curve. Thus, the processing at the integral value calculating section 40 is completed. 

so [0121] It is to be noted that, at the integration executing section 43, the integration along the Integral path delivered 
from the integral path determining section 41 is carried out with respect to, e.g., segment (line segment) (hereinafter 
referred to as path segment as the occasion may demand) between the point where the key signal a becomes equal 
to 0 (the point p^ and the point where it becomes equal to 1 (the point p 2 ) of that Integral path, i.e., the integral path 
within the gray area. This is because since key signals a in areas except for the gray area take any one of values 0 

55 and 1 , it is unnecessary to determine a with respect to such areas in accordance with the equation (1 8). Accordingly, 
only the key signal a on the path segment is calculated at the integration executing section 43, and the above-described 
skeleton portion means the path segments in all integral paths delivered from the integral path determining section 41 . 
[0122] The skeleton portion of the key signal, and the P1 closed curve and the P2 closed curve obtained at the 
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integral value calculating section 40 In a manner as described above are delivered to the key signal generating section 
50. 

[0123] At step S6 (FIG. 1 1 ), the key signal generating section 50 carries out interpolation with respect to the skeleton 
portion of the key signal to generate a key signal a corresponding to the entirety of the picture C. Thus, the processing 

5 at the key signal generating section 50 is completed. 

[0124] A more practical example of the configuration of the key signal generating section 50 is shown in FIG. 27. An 
interpolating section 51 for inside of the gray area is supplied with the skeleton portion of the key signal from the integral 
value calculating section 40 and key signals on the P1 closed curve and the P2 closed curve from 0/1 allocating section 
52 which will be described later. Thus, the interpolating section 51 for inside of the gray area carries out interpolation 

10 of the key signal In the area except for the skeleton portion, the P1 closed curve and the P2 closed curve of the gray 
area shown in FIG. 28A in such a manner that slanting lines are attached (the area surrounded by the skeleton portion, 
the P1 closed curve and the P2 closed curve). 

[0125] Namely, FIG. 28A shows xya space in which picture C Is disposed on the xy-plane and axis (ct-axis) perpen- 
dicular to that xy-plane is taken. It is now assumed that key signals at respective points on the picture C are plotted in 

is the a-axis direction within this xya space. Since the key signals delivered to the interpolating section 51 for the inside 
of the gray area are the skeleton portion of the key signal (key signal on the path segment), key signals on the P1 
closed curve (since the P1 closed curve is the curve in which points p 1 where a is equal to 0 are connected, values of 
key signals thereat are 0 (zero)), and key signals on the P2 closed curve (since the P2 closed curve is the curve in 
which points p 2 where a is equal to 1 are connected, values of key signals thereat are 1), points corresponding to the 

20 skeleton portion of the key signal are plotted on the path segment, and points of a = 0 and a = 1 are respectively plotted 
on the P1 closed curve and the P2 closed curve. Thus, a frame as indicated by thick line in FIG. 28A (hereinafter 
referred to as the skeleton frame as the occasion may demand) is formed on the xya space. 
[0126] Then, the interpolating section 51 for the inside of the gray area generates curved surface Q for interpolating 
the inside of the skeleton frame to allow a that this curved surface Q provides (designates) to be key signals a at 

25 respective points within the area formed when the curved surface Q is projected onto the xy-plane (portions to which 
slanting lines are attached in FIG. 28B which will be referred to as patch areas as the occasion may demand). 
[0127] In this case, the interpolating section 51 for the inside of the gray area carries out, as shown in FIG. 288, 
interpolation of key signals as described above with respect to the areas surrounded by the P1 and P2 closed curves 
and the path segment, i.e., all of the patch areas. 

30 [0128] The skeleton portion of the key signal, the key signals (a = 0) on the P1 closed curve, the key signals (a = 1 ) 
on the P2 closed curve, and the key signals In the patch area obtained by interpolation are delivered from the interpo- 
lating section 51 for the inside of the gray area to a synthesis section 54. 

[01 29] The 0/1 allocating section 52 is supplied with the P1 closed curve and the P2 closed curve. The 0/1 allocating 
section 52 respectively allocates 0 and 1 as key signals a on the P1 closed curve and the P2 closed curve to deliver 

35 them to the interpolating section 51 for the inside of the gray area. Moreover, an interpolating section 53 for area except 
for the gray area is supplied with the P1 closed curve and the P2 closed curve. As shown in FIG. 28B, the interpolating 
section 53 for area except for the gray area interpolates 0 as key signals at respective points within the area put between 
the picture frame of the picture C and the P1 closed curve (the area considered to be completely background B, which 
will be referred to as a0 area as the occasion may demand hereinafter), and interpolates 1 as key signals at respective 

40 points within the area surrounded by the P2 closed curve (the area considered to be completely the foreground F, 
which will be referred to as a1 area as the occasion may demand hereinafter), thus to deliver them to the synthesis 
section 54, 

[0130] The synthesis section 54 synthesizes (arranges (allocates) at corresponding positions within the picture frame 
of the picture C) the skeleton portion of the key signal, the key signals on the P1 closed curve, the key signals on the 
45 P2 closed curve, and the key signals of the patch area obtained by interpolation from the interpolating section 51 for 
the inside of the gray area, and the key signals of the a0 area and the key signals of the a1 area from the interpolating 
section 53 for area except for the gray area, thereby to generate a key signal corresponding to the entirety of the picture 
C. 

[0131] The operation of the key signal generating section 50 will now be described with reference to the flowchart 
so of FIG. 29. 

[0132] At step S5-1 , the 0/1 allocating section 52 respectively allocates 0, 1 to the P1 closed curve and the P2 closed 
curve to generate key signals on the P1 closed curve and key signals on the P2 closed curve. The key signals on the 
P1 closed curve and the key signals on the P2 closed curve are delivered to the interpolating section 51 for the inside 
of the gray area. 

55 [0133] At step S5-2, the interpolating section 51 for the inside of the gray area is operative so that when the key 
signals on the P1 closed curve and the key signals on the P2 closed curve are delivered thereto from the 0/1 allocating 
section 52, it forms the skeleton frame as has been explained with reference to FIG. 28A to further generate curved 
surface (interpolation curved surface) Q for interpolating the Inside of the skeleton portion. 
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[0134] At step S5-3, the interpolating section 51 for the inside of the gray area generates key signals of the patch 
areas on the basis of the curved surface Q to deliver them to the synthesis section 54. 

[0135] At step S5-4, the interpolating section 53 for area except for the gray area interpolates 0 as key signals at 
respective points within the ctO area, and interpolates 1 as key signals at respective points within the a1 area, thus to 

5 deliver them to the synthesis section 54. 

[0136] At step S5-5, the synthesis section 54 synthesizes the skeleton portion of the key signal, the key signals on 
the P1 closed curve, the key signals on the P2 closed area and the key signals of the patch areas obtained by inter- 
polation delivered from the interpolating section for the inside of the gray area, and the key signals of the aO area and 
the key signals of the a1 area delivered from the interpolating section 53 for area except for the gray area to generate 

10 a key signal corresponding to the entirety of the picture C. Then, the synthesis section 54 converts this key signal 
(continuous key signal) into key signals of pixel units by implementing, e.g., filtering, etc. thereto as previously described 
to output these key signals as a key signal for extracting the foreground F from the picture, and allowing the extracted 
foreground F to undergo synthesis (keying) with respect any other picture. Thus, the processing at the key signal 
generating section 50 is completed. 

is [0137] It is to be noted that the processing of the step S5-4 may be carried out prior to the processing of the steps 
S5-1 to S5-3, or may be carried out at the same time (in parallel). 

[0138] Further, while the key signals of the patch areas are obtained from the curved surface Q for interpolating the 
skeleton frame in the above-described case, the key signals of the patch areas may be determined, in addition to the 
above, e.g., in a manner as described below. Namely, an approach may be employed to determine key signals at points 

20 adjacent to the path segment, the P1 closed curve and the P2 closed curve within the gray area on the basis of the 
skeleton portion of the key signal, the key signals on the P1 closed curve and the key signals on the P2 closed curve 
which have been already obtained to sequentially, at times subsequent thereto, key signals at points adjacent to points 
where the key signals have been determined on the basis of already determined key signals. 
[0139] Moreover, while such an approach is employed in the above-described case to determine key signals of the 

25 patch areas from the curved surface Q for interpolating the skeleton frame formed by plotting points corresponding to 
the skeleton portion of the key signal onto the path segment, and respectively plotting respective points of a = 0, a = 
1 onto the P1 closed curve and the P2 closed curve, the key signals of the patch areas may be determined, in addition 
to the above-mentioned approach, e.g., from a curved surface for interpolating frame obtained by further plotting points 
of a = 0.5 as well onto the path determination closed curve. 

30 [0140] While the path determination closed curve, i.e., boundary between the foreground F and the background B 
is determined from change C of the pixel value in the embodiment of FIG. 10, the path determination closed curve 
may be determined on the basis of instruction from users of the apparatus. 

[0141] A more practical example of the configuration of a key signal generating apparatus for calculating a path 
determination closed curve on the basis of instruction from a user is shown in FIG. 30. It is to be noted that the same 

35 reference numerals are respectively attached to portions corresponding to the key signal generating apparatus shown 
in FIG. 10, and the explanation will be omitted as the occasion may demand in the following description. 
[0142] Namely, at this key signal generating apparatus, a feature point detecting section 60 is provided in place of 
the feature point detecting section 20, and an operation section 1 is newly provided. In addition, other portions of this 
key signal generating apparatus are respectively constituted in the same manner as in the key signal generating ap- 

40 paratus shown in FIG. 10. 

[0143] The operation section 1 is composed of, e.g., tablet and pen or mouse, etc., and is operated when a user 
designates the boundary between the foreground F and the background B while looking at picture C displayed on 
display (not shown), in addition, an operation signal corresponding to that operation is delivered to the feature point 
detecting section 60. 

45 [0144] The feature point detecting section 60 detects the path determination closed curve and the feature points on 
the basis of the operation signal from the operation section 1 . A more practical example of the configuration of the 
feature point detecting section 60 is shown in FIG. 31 . It is to be noted that, in the feature point detecting section 60 
shown in FIG. 31, the same reference numerals are respectively attached with respect to portions corresponding to 
the feature point detecting section 20 shown in FIG, 16 and their explanation will be omitted as the occasion may 

so demand in the following description. Namely, a line thinning section 61 is provided, in place of the boundary detecting 
section 21 , at the feature point detecting section 60, and other portions of the feature point detecting section 60 are 
constituted in the same manner as in the feature point detecting section 20. 

[0145] The line thinning section 61 recognizes line designated by user (hereinafter referred to as designation line as 
the occasion may demand), which corresponds to the boundary between the foreground F and the background B, on 
55 the basis of the operation signal delivered from the operation section 1 to carry out line thinning to thereby detect a 
path determination closed cu rve. For example, the designation line Is displayed on a display where picture C is displayed 
in a manner overlaid upon the picture C, and is line having width of one pixel or more as shown in FIG. 32A. In view 
of this, the line thinning section 61 determines, as shown in FIG. 32B, for example, in order to determine path deter- 
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mination closed curve as an ideal line having no width from such designation line having width, a line (curve) which 
minimizes square error of distances from the centers of pixels constituting the designation line (portions indicated by 
mark x in FIG. 32B) by the least square (method) to allow this line to be the path determination closed curve. 
[0146] This path determination closed curve is delivered from the line thinning section 61 to the feature point deter- 
s mining section 22. 

[0147] It is to be noted that the method of thinning the designation line is not limited to the above-described method, 
but other methods may be used for this purpose. Namely, e.g., an approach may be employed to remove pixels con- 
stituting the designation line from pixels positioned outside to thereby generate a thin line having width consisting of 
one pixel to further determine a line which most satisfactorily approximate a line passing through the centers of pixels 

10 constituting that thin line to allow this line to be the path determination closed curve. 

[0148] In the key signal generating apparatus constituted as described above, when designation line corresponding 
to the boundary between the foreground F and the background B (pixels constituting the boundary) as the result of the 
fact that the operation section 1 is operated by user, thinning of that designation line is carried out at the line thinning 
section 61 of the feature point detecting section 60, whereby the path determination closed curve is determined. The 

15 path determination closed curve thus obtained is delivered to the feature point determining section 22. At times sub- 
sequent thereto, processing similar to the key signal generating apparatus shown in FIG. 10 is carried out. Thus, a 
key signal a corresponding to the entirety of the picture C is determined. 

[0149] Accordingly, since the path determination closed curve is determined on the basis of designation by user 
(designation line) in this case, change C of the pixel value becomes unnecessary in order to determine the path 
20 determination closed curve. Namely, in this case, there is no necessity of delivering change C of the pixel value with 
respect to the feature point detecting section 60 from the change detecting section 1 0. 

[0150] While the key signal generating apparatus shown in FIG. 10 is adapted to determine difference F-B at the 
F-B detecting section 30 to normalize the change C of the pixel value thereafter to determine the key signal, the key 
signal generating apparatus may be of construction in which no F-B detecting section 30 is provided, 

25 [01 51 ] As a more practical example, the configuration of another embodiment of the key signal generating apparatus 
to which this invention is applied is shown in FIG. 33. It is to be noted that, in the key signal generating apparatus 
shown in FIG. 33, the same reference numerals are respectively attached with respect to portions corresponding to 
the key signal generating apparatus shown in FIG. 10,-and their explanation will be omitted as the occasion may 
demand. Namely, this key signal generating apparatus is adapted so that the F-B detecting section 30 is omitted and 

30 an integral value calculating section 70 is provided in place of the integral value calculating section 40. In addition, 
other portions of the key signal generating apparatus are constituted in the same manner as in the key signal generating 
apparatus shown in FIG. 1 0. 

[0152] The integral value calculating section 70 is operative so that, within the range where quantity (magnitude) of 
change C of the pixel value is above a predetermined threshold value K (hereinafter referred to as integral range as 

35 the occasion may demand), it integrates its change quantity, e.g., along the direction in which change of the pixel value 
C has been detected, or the like to calculate a key signal (the skeleton portion of the key signal). 
[0153] Namely, the integral value calculating section 70 first determines points on the boundary of the gray area, i. 
e., points p 1( p 2 where a is equal to 0, 1 of points on, e.g., the maximum inclination (gradient) direction path U (It is to 
be noted that the path itself may be path except for the maximum inclination (gradient) direction path U as described 

40 above). In more practical sense, as shown in FIG. 34A, for example, the integral value calculating section 70 determines 
two points where magnitude of change C of the pixel value delivered from the change detecting section 10 becomes 
equal to the predetermined threshold value Kto allow them to be respectively points p, , p 2 (for example, when observing 
the change C of the pixel value from the background B side toward the foreground F side, the point where the change 
C of the pixel value becomes equal to the threshold value K in the beginning is caused to be the point p^ , and the point 

45 where the change C of the pixel value becomes equal to the threshold value K thereafter is caused to be the point p 2 .). 
[0154] Further, the integral value calculating section 70 calculates, in accordance with the following equation (19), 
integral value (definite integral value) D (definite integral information) of change of the pixel value C within the range 
(integral range) from the point p 1 to the point p 2 , e.g., along the maximum inclination (gradient) direction path U. 

50 

D ■ fc'lp)dp ••• (19) 

pi 

55 

[0155] Then, the integral value calculating section 70 normalizes the change C of the pixel value by the definite 
integral value D to thereby determine normalized value C'/D to integrate this normalized value C'/D, e.g., in accordance 
with the following equation (20) to thereby determine a key signal a with respect to the pixel at the point (position) p. 
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[0156] The definite integral value D corresponds to the absolute value !f-b! of difference F-B in the case of the key 
signal generating apparatus shown in FIG. 1 0. Accordingly, when the change C of the pixel value is normalized by the 
definite integral value D to carry out the integration by the equation(20), it is possibte to easily obtain a key signal a 

10 which takes value within the range from 0 to 1 . It is to be noted that while, in this case, the key signal a is obtained 
within the range from the points (positions) p1 to p2 (the portion labeled "effective (valid) width" in FIG. 34B), values 
of key signals within the range at the left side (positioned) relative to the point p 1 or the range at the right side (position ed) 
relative to the point p 2 in FIG. 34B are caused to be respectively 0 and 1 (interpolation of 0 and 1 is carried out at the 
key signal generating section 50 as described above). 

is [0157J Since it is unnecessary to provide the F-B detecting section 30 at this key signal generating apparatus from 
facts as described above, the apparatus can be of a smalier-size configuration as compared to the key signal generating 
apparatus shown in FIG. 10. 

[0158] The operation of the key signal generating apparatus shown in FIG. 33 will now be described with reference 
to the flowchart shown in FIG. 35. 
20 [01 59] At step S21 , the change detecting section 1 0 calculates change C of the pixel value between pixels of picture 
(input picture) C similarly to the case of the step S1 of FIG. 1 1 to deliver it to the feature point detecting section 20 and 
the integral value calculating section 70. 

[0160] At step S22, the feature point detecting section 20 is operative so that when the change C of the pixel value 
is delivered thereto from the change detecting section 1 0, it detects the path determination closed curve and the feature 
25 points similarly to the case at the step S2 of FIG. 11 to deliver the path determination closed curve and the feature 
points to the integral value calculating section 70. 

[0161] At step S23, the integral value calculating section 70 is operative so that the change C of the pixel value is 
delivered thereto from the change detecting section 1 0 and the path determination closed curve and the feature points 
are delivered thereto from the feature point detecting section 20, it determines the integral path, e.g., similarly to the 

30 integral value calculating section 40 of FIG. 1 0 to further determine points p v p 2 where magnitude of change C of the 
pixel value becomes equal to a predetermined threshold value K, as has been explained with reference to FIG. 34A, 
on the integral path. Then, the integral value calculating section 70 calculates integral value D of the quantity of the 
change C of the pixel values at the points on the integral path existing within the range from the point to the point 
p 2 in accordance with the equation (19). 

35 [01 62] At step S24, the integral value calculating section 70 normalizes the change C of the pixel value by the integral 
value D to calculate normalized value C'/D. 

[0163] At step S25, the integral value calculating section 70 determines, in accordance with the equation (20), the 
key signal with respect to the points p constituting the integral path (path segment), i.e., the skeleton portion of the key 
signal to deliver it to the key signal generating section 50. Further, the integral value calculating section 70 generates 
40 pi closed curve and P2 closed curve similarly to the integral value calculating section 40 of FIG. 1 0 to deliver them to 
the key signal generating section 50. 

[0164] At step S26, the key signal generating section 50 carries out processing similar to that in the case of the step 
S6 of FIG. 11 to thereby generate a key signal a corresponding to the entirety of the picture C. Thus, the processing 
at the key signal generating apparatus is completed. 

45 [0165] An example of the configuration of the integral value calculating section 70 of FIG. 33 is shown in FIG. 36. 
[0166] Path determination closed curve and feature points from the feature point detecting section 20 are delivered 
to an integral path determining section 71 , and change C of the pixel value from the change detecting section 10 is 
delivered to the integral path determining section 71 , a definite integral value calculating section 72, and a normalization 
executing section 73. In this example, the integral path determining section 71 is also supplied with a predetermined 

so threshold value K from circuit (not shown). 

[0167] The integral path determining section 71 determines integral paths passing through respective feature points, 
similarly to the integral path determining section 41 shown in FIG. 24, on the basis of the feature points and the path 
determination closed curves to deliver these integral paths to the definite integral value calculating section 72, the 
normalization executing section 73 and the integration executing section 74. Further, the integral path determining 

55 section 71 determines points where the magnitude of the change C of the pixel value becomes equal to the predeter- 
mined threshold value K, i.e., points p 1( p 2 of points on the Integral paths passing through respective feature points to 
deliver these points p.,, p 2 to the definite integral value calculating section 72, an integration executing section 74 and 
a closed curve generating section 75. 
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[0168] The definite integral value calculating section 72 performs an operation in accordance with the equation (1 9). 
Namely, the definite integral value calculating section 72 carries out definite integral of change C of the pixei value 
within the range from the point p, up to the point p 2 similarly delivered from the integral path determining section 71 
along the integral paths passing through respective feature points delivered from the integral path determining section 
5 71 to deliver, to the normalization executing section 73, integral values (definite integral values) D every respective 
integral paths obtained as the result thereof. 

[01 69] The normalization executing section 73 normalizes change C of the pixel value on each integral path delivered 
from the integral path determining section 71 by the definite integral value D with respect to each integral path delivered 
from the definite integral value calculating section 72 to deliver the normalized value C7D obtained as the result thereof 

io to the integration executing section 74. 

[0170] The integration executing section 74 performs an operation in accordance with the equation (20). Namely, 
the integration executing section 74 carries out integration of the normalized value C7D delivered from the normalization 
executing section 73 along each integral path delivered from the integral path determining section 72 to output the 
skeleton portion of the key signal as the integral value on each integral path (path segment) obtained as the result 

is thereof. It is to be noted that while the integration along each integral path at the integration executing section 74 is 
carried out according to the equation (20) in such a manner that the point p 1 on each Integral path delivered from the 
integral path determining section 71 is caused to be the initial (start) point, such integration may be also carried out in 
such a manner that the point p 2 is caused to be the initial point. 

[0171] The closed curve generating section 75 is operative so that when points p 1t p 2 on all integral paths are de- 
20 livered thereto from the integral path determining section 71 , it connects the points p., to each other, and connects the 
points p 2 to each other to thereby respectively generate P1 closed curve and P2 closed curve. 
[01 72] The operation of the integral value calculating section 70 will now be described with reference to the flowchart 
of FIG. 37. 

[0173] At step S21-1 , the integral path determining section 71 carries out processing similar to that in the case of 
25 the step S4-1 of FIG. 26 to determine integral paths passing through the remarked feature points to deliver these 
integral paths to the definite integral value calculating section 72, the normalization executing section 73 and the inte- 
gration executing section 74. 

[0174] At step S21-2, the integral path determining section 71 detects points p v p 2 where the magnitude of change 
C of pixel value delivered from the change detecting section 10 becomes equal to the predetermined threshold value 

30 K of points on the integral paths passing through the remarked feature points to deliver these points p^ , p 2 to the definite 
integral value calculating section 72, the integration executing section 74 and the closed curve generating section 75. 
[0175] At step S21-3, the definite integral value calculating section 72 is operative so that when the integral path 
with respect to the remarked feature point and points p, , p 2 on the integral path are delivered thereto from the integral 
path determining section 71 , it carries out integration of change C of the pixel value within the range from the point p A 

35 to the point p 2 along the integral path in accordance with the equation (19) to deliver definite integral value D obtained 
as the result thereof to the normalization executing section 73. 

[0176] At step S21 -4, the normalization executing section 73 normalizes the change C of the pixel value of pixel on 
the integral path with respect to the remarked feature point by the definite integral value D delivered from the definite 
integral value calculating section 72 to deliver normalized value C7D obtained as the result thereof to the integration 
40 executing section 74. 

[0177] At step S21-5, the integration executing section 74 integrates, in accordance with the equation (20), the nor- 
malized value C7D delivered from the normalization executing section 73, with, e.g., the point p^ delivered from the 
integral path determining section 71 being as the initial (start) point, along the integral path similarly delivered from the 
integral path determinating section 71 to calculate the key signal on the integral path, i.e., the skeleton portion of the 
45 key signal. 

[0178] At step S21-6, the integral path determining section 71 judges whether or not all of feature points from the 
feature point detecting section 20 have been caused to be the remarked feature point. In the case where the integral 
path determining section 71 has judged at this step S21-6 that all of feature points from the feature point detecting 
section 20 have not yet been caused to be the remarked feature point, the processing returns to the step S21 -1 . Thus, 

50 the integral path determining section 71 allows the feature point which is not yet caused to be the remarked feature 
point to be newly the remarked feature point At times subsequent thereto, processing as described above is repeated. 
[0179] On the other hand, in the case where the integral path determining section 71 has judged at the step S21 -6 
that ail of feature points have been caused to be the remarked feature point, i.e., integration of the equation (20) along 
all the integral paths passing through respective feature points delivered from the feature point detecting section 20, 

55 whereby key signals on those integral paths (the skeleton portions of the key signals) have been determined, the 
processing proceeds to step S21 -7. 

[0180] At the step S21-7, the closed curve generating section 75 generates P1 closed curve and P2 closed curve. 
Thus, the processing at the integral value calculating section 70 is completed. 
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[0181] Namely, the closed curve generating section 75 connects the points p<, to each other and connects the points 
p 2 to each other, those points existing on all integral paths delivered from the integral path determining section 71 to 
thereby respectively generate P1 closed curve and P2 closed curve. 

[0182] It is to be noted that the integration at the integration executing section 74 is carried out only with respect to 
the path segment of the integral path similarly to the case at the integration executing section 43 shown in FIG. 24. 
[0183] FIG. 38 shows the configuration of an embodiment of a picture synthesis apparatus to which this invention is 
applied. 

[0184] Picture providing units comprise, e.g., a video tape recorder, hard disc device or disc player, etc., and serve 
to reproduce a predetermined picture to output it. In this example, picture T1 (first picture) in which the foreground and 
the background that the picture providing unit 81 reproduces are caused to be respectively F and B is delivered to a 
key signal generating unit 84 and a mixer 85. Thus, this picture T1 is caused to be the object picture for generating 
the key signal. Moreover, picture T2 (second picture) that the picture providing unit 82 reproduces is delivered to the 
mixer 85, at which the foreground F of the picture T1 is synthesized with respect to the picture T2. Further, picture T3 
obtained by photographing or picking up image of only the background B of the picture T1 that the picture providing 
unit 81 reproduces is outputted from the picture providing unit 83. Namely, when, e.g., the picture T1 is assumed to 
be the picture image where a person is walking within a certain scene, the picture T3 in which image of that scene 
(background B) is picked up in the state where that person (foreground F) is excluded is outputted from the picture 
providing unit 83. The picture T3 that the picture providing unit 83 reproduces is delivered to the mixer 85. 
[0185] The key signal generating unit 84 is constituted in a manner similar to the key signal generating unit of FIG. 
1 0, 30 or 33, and generates a key signal a as described above to deliver it to the mixer 85. The mixer 85 synthesizes 
the picture T1 and the picture T2 inputted thereto by using the key signal a from the key signal generating unit 84. 
Namely, the mixer 85 extracts the foreground F from the picture T1 to synthesize (incorporate) that foreground F into 
(with respect to) the picture T2. 

[0186] The operation of the picture synthesis apparatus will now be described. 

[0187] At the picture providing units 81 to 83, pictures T1 to T3 are respectively reproduced. These pictures T1 to 
T3 are delivered to the mixer 85. Further, the picture T1 that the picture providing unit 81 reproduces is delivered also 
to the key signal generating unit 84. 

[0188] The key signal generating unit 84 generates a key signal a in a manner as described above to deliver it to 
the mixer 85. The mixer 85 extracts the foreground F from the picture T1 inputted thereto by using the key signal a to 
further synthesize that foreground F Into (with respect to) the picture T2 to generate picture in which the foreground F 
is synthesized into (with respect to) the picture T2 (synthetic picture) T4. 

[0189] Namely, assuming now that pixel values of pixels at respective positions p constituting respective pictures 
T1 , T2 are respectively designated at P(p), Q(p), pixel values of pixels at respective positions p constituting respective 
foreground and background of the picture T1 are respectively designated at F(p), B(p), and pixel value of pixel at each 
position p constituting the synthetic picture T4 is designated at R(p), the following equations (relational expressions) 
(21), (22) hold: 

P(p) = aF(p) + (1-a)B(p) (21) 



R(p) = aF(p) + (1-a)Q(p) (22) 

[0190] When aF(p) of the right side of the equation (22) is eliminated from the equation (21), the equation (22) is 
rewritten into the following equation (23) 

R(p) = P(p) - (1 - a)B(p) + (1 -a)Q(p) (23) 

[0191] At the mixer 85, synthetic picture T4 is generated in accordance with the equation (23). Namely, pixel values 
P(p), Q(p), B(p) are respectively delivered from the picture providing units 81 to 83 to the mixer 85, and the key signal 
cc(p) is delivered from the key signal generating unit 84 also to the mixer 85. At the mixer 85, operation of the equation 
(23) is performed by using the respective pixel values and the key signal. Thus, picture T4 consisting of pixel value R 
(p), i.e., in which the foreground F is synthesized with respect to the picture T2 is generated. 
[0192] This picture T4 is delivered to, e.g., recording unit (e.g., video tape recorder, hard disc device, or writable disc 
drive, etc.), at which it is recorded. 

[0193] Since precise key signals, i.e., key signals in which the influence of motion blur Is taken into consideration 
and alias is permitted to be reduced are outputted from the key signal generating unit 84 as described above, in the 
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case where such key signals are used to carry out synthesis of picture, precise synthetic picture (picture free from 
disagreement of feeling when viewed) can be obtained. 

[0194] It is to be noted that the picture synthesis apparatus shown in FIG. 38 may be constituted as described below. 

Namely, since background B is provided by the picture providing unit 83, in the case where the key signal generating 
5 unit 84 is constituted by such a key signal generating apparatus as shown in FIG. 1 0 or 30 including the F-B detecting 

section 30, background B is delivered to the key signal generating unit 84 as indicated by dotted lines in FIG. 38, thus 

making it possible to determine difference F-B from the background B in the key signal generating unit 84. 

[0195] Meanwhile, since picture T2 consisting of pixel value Q(p) is obtained from the picture providing unit 82 and 

the key signal a is obtained from the key signal generating unit 84, if the pixel value F(p) of the foreground F can be 
10 recognized (the pixel value F(p) can be estimated), it is possible to generate, also by the equation (22), picture T4 in 

which the foreground F is synthesized with respect to the picture T2. 

[0196] In view of the above, the configuration of an embodiment of the picture synthesis apparatus adapted for 
generating picture T4 in accordance with the equation (22) is shown in FIG. 39. It is to be noted that the same reference 
numerals are respectively attached with respect to portions corresponding to the picture synthesis apparatus shown 
is in FIG. 38 in the picture synthesis apparatus shown in FIG. 39, and their explanation will be omitted as the occasion 
may demand. Namely, In this picture synthesis apparatus, an information generating unit 90 and a mixer 96 are provided 
in place of the picture providing unit 83 and the mixer 85. In addition, other portions of the picture synthesis apparatus 
are constituted similarly to the picture synthesis apparatus of FIG. 38. 

[0197] The information generating unit 90 is supplied with picture T1 that the picture providing unit 81 outputs. The 
20 information generating unit 90 estimates (generates), from the picture T1 , pixel value F(p) of the foreground F thereof 
to deliver it to the mixer 96. The mixer 96 performs an operation in accordance with the equation (22) by using pictures 
T1 , T2 (pixel values P(p), Q(p)) that the picture information providing units 81 , 82 respectively output, key signal a(p) 
that the key signal generating unit 84 outputs, and foreground F (pixel value F(p)) that the information generating unit 
90 outputs to thereby generate synthetic picture T4 in which the foreground F is synthesized with respect to the picture 
25 T2. 

[0198] In the picture synthesis apparatus constituted as described above, pictures T1 , T2 are respectively reproduced 
at the picture providing units 81 , 82. These pictures T1 , T2 thus reproduced are delivered to the mixer 96. Further, the 
picture T1 that the picture providing unit 81 reproduces is delivered also to the information generating unit 90 and the 
key signal generating unit 84. The key signal generating unit 84 generates the key signal a in a manner as described 
30 above to deliver it to the mixer 96. 

[0199] On the other hand, the information generating unit 90 determines, from the picture T1 , pixel value F(p) of the 
foreground F thereof. 

[0200] A more practical example of the configuration of the information generating unit 90 is shown in FIG. 40. This 
information generating unit 90 is composed of a change detecting section 91 , a boundary detecting section 92, a very 

35 small area detecting section 93, a color distribution forming section 94, and a center of gravity detecting section 95. 
The change detecting section 91 is constituted similarly to the change detecting sections 10 respectively shown in 
FIGS. 1 0, 30 and 33, and the boundary detecting section 92 is constituted similarly to the boundary detecting section 
21 of FIG. 16. Moreover, the very small area detecting section 93, the color distribution forming section 94 and the 
center of gravity detecting section 95 are constituted respectively similarly to the very small area detecting section 31 , 

40 the color distribution forming section 32F and the center of gravity detecting section 33F. It is to be noted that the very 
small area detecting section 93 extracts only pixels constituting the foregrounds F with respect to respective very small 
areas to deliver pixel values of those pixels, i.e. , only F boundary pixel values to the color distribution forming section 94. 
[0201] Accordingly, at this information generating unit 90, change P' of the pixel value P of picture T1 is determined 
at the change detecting section 91 , and path determination curve and feature points are determined on the basis of 

45 the change P* at the boundary detecting section 92. Further, at the very small area detecting section 93, very small 
area including respective feature points is detected, and F boundary pixel values are extracted with respectto respective 
very small areas. Further, color distribution on the color space of F boundary pixel values in respective very small areas 
is determined at the color distribution forming section 94. In addition, center of gravity of the color distribution is deter- 
mined at the center of gravity detecting section 95, and the center of gravity thus obtained is outputted as pixel value 

so F(p) of the foreground F of pixels existing at the periphery of respective feature points in the gray area. 

[0202] In this example, at the information generating unit 90, with respect to the portion except for the gray area of 
the foreground F (completely the portion of the foreground F), pixel value F(p) of the foreground F thereof is outputted 
as It is. 

[0203] Turning back to FIG. 39, at the mixer 96, pixel value R(p) of picture T4 in which the foreground F is synthesized 
55 with respect to the picture T2 is determined in accordance with the equation (22) by using pixel values P(p), Q(p) that 
the picture information providing units 81 , 82 respectively output, key signal <x(p) that the key signal generating unit 84 
outputs, and pixel value F(p) that the information generating unit 90 outputs. 
[0204] Accordingly, also in this case, it is possible to obtain precise synthetic picture. 
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[0205] In this instance, at the information generating unit 90, background B may be generated in place of generating 
foreground F. In this case, at the mixer 96, pixel value R(p) of synthetic picture T4 may be determined in accordance 
with the equation (23) by using the background B generated by the information generating unit 90. 
[0206] It is to be noted that the synthesis method for picture at the mixers 85, 96 are not limited to the above-described 
method, but other synthesis methods may be used. 

[0207] Further, while integration is carried out at the integral value calculating sections 40, 70 in this embodiment, 
there may be employed, in place of the integration, an operation equivalent to the integration, I.e., an operation to set 
sample points at the path for carrying out Integration to add sample values at respective sample points. 
[0208] In addition, while key signal portion except for the skeleton portion of key signals in the gray area is determined 
by interpolation in this embodiment, key signal portion except for the skeleton portion of the key signal may be deter- 
mined, e.g., by carrying out integration based on the equation (18) or the equation (20) In place of the interpolation. 
[0209] It should be noted that various modifications and/or application examples are conceivable (applicable) within 
the range which does not depart from the gist of this invention. Accordingly, the gist of this invention is not limited to 
the embodiments. 

Industrial Applicability 

[0210] In accordance with the key signal generating apparatus and the key signal generating method according to 
this invention, change of pixel value between pixels constituting picture is detected to generate a key signal on the 
basis of change of that pixel value. Accordingly, precise key signal can be obtained. 

[0211] In accordance with the picture synthesis apparatus and the picture synthesis method according to this inven- 
tion, change of pixel value between pixels constituting a first picture is detected to generate a key signal on the basis 
of the change of that pixel value. Then, that key signal is used to synthesize the foreground picture of the first picture 
and the background picture of a second picture. Accordingly, precise synthetic picture can be obtained. 



Claims 

1. A key signal generating apparatus adapted for generating a key signal from a picture, 

the apparatus comprising: 

change detecting means for detecting a change of pixel value between pixels constituting said picture, and 
outputting change data expressing the distance on a color space between pixels close to each other of said 
picture; and 

key signal generating means for generating said key signal on the basis of said change data integrated along 
a predetermined integral path in a direction in which a change of the pixel value has been detected by the 
change detecting means. 

2. A key signal generating apparatus as set forth in claim 1 , wherein the key signal generating means integrates said 
change data along said predetermined integral path within an integral range which is the range where said change 
data takes a predetermined threshold value or more to thereby generate the key signal. 

3. A key signal generating apparatus as set forth in claim 1 or 2, wherein 

the picture consists of foreground picture and background picture , and wherein 

the apparatus further comprises difference detecting means for detecting difference between the foreground 
picture and the background picture to output difference data indicating that difference. 

4. A key signal generating apparatus as set forth in any one of claims 1 to 3, wherein the key signal generating means 
comprises: 

normalizing means for normalizing the change data by the difference data to output normalized data obtained 
as the result of the normalization, and 

integration executing means for integrating the normalized data along said predetermined integral path. 

5. A key signal generating apparatus as set forth in any one of claims 2 to 4, wherein the key signal generating means 
comprises: 
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range detecting means for detecting the integral range; 

definite integral information generating means for integrating the change data along said predetermined inte- 
gral path within the integral range to generate definite integral information of the change data with in the integral 
range; 

normalizing means for normalizing the change data by the definite Integral information to output normalized 
information obtained as the result of the normalization; and 

integration executing means for integrating the normalized information to thereby generate the key signal. 

6. A picture synthesis apparatus adapted for receiving a first picture consisting of at least foreground picture and. 
background picture , and a second picture consisting of at least background picture to synthesize the foreground 
picture of the first picture and the background picture of the second picture, 

the apparatus comprising: 

change detecting means for detecting a change of pixel value between pixels constituting said first picture, 
and outputting change data expressing the distance on a color space between pixels close to each other of 
said first picture; 

key signal generating means for generating said key signal on the basis of said change data integrated along 
a predetermined integral path in a direction in which a change of the pixel value has been detected by the 
change detecting means; and 

synthesis means for synthesizing the foreground picture of the first picture and the background picture of the 
second picture by using the key signal generated by said key signal generating means. 

7. A picture synthesis apparatus as set forth in claim 6, which further comprises difference detecting means for de- 
tecting difference between the foreground picture and the background picture of the first picture to output difference 
information indicating the difference. 

8. A picture synthesis apparatus as set forth in claim 6 or 7, wherein the key signal generating means integrates the 
change data within an integral range which is the range where the change information takes a predetermined 
threshold value or more. 

9. A key signal generating method for generating a key signal from a picture, 

the method comprising the steps of: 

detecting a change of pixel value between pixels constituting said picture, and outputting change data ex- 
pressing the distance on a color space between pixels close to each other of said picture is calculated; and 
generating said key signal on the basis of said change data integrated along a predetermined integral path in 
a direction in which a change of the pixel value has been detected by said detecting step. 

10. A key signal generating method as set forth in ciaim 9, wherein said change data are integrated along said pre- 
determined integral path within an integral range which is the range where said change data takes a predetermined 
threshold value or more to thereby generate the key signal. 

11. A key signal generating method as set forth in claim 9 or 10, 

wherein the picture.consists of foreground picture and background picture, 
the method comprising the step of 

detecting difference between the foreground picture and the background picture to output difference data 
indicating that difference. 

12. A key signal generating method as set forth in any one of claims 9 to 11 , the method comprising the steps of: 

normalizing the change data by the difference data to output normalized data obtained as the result of the 
normalization, and 

integrating the normalized data along said predetermined integral path. 

13. A key signal generating method as set forth in any one of claims 10 to 12, wherein the method comprising the 
steps of: 
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detecting the integral range; 

integrating the change data along said predetermined integral path within the integral range to generate definite 
integral information of the change data within the integral range; 

normalizing the change data by the definite integral information to output normalized information obtained as 
the result of the normalization; and 

integrating the normalized information to thereby generate the key signal. 

14. A picture synthesis method of receiving a first picture consisting of at least foreground picture and background 
picture, and a second picture consisting of at least background picture to synthesize the foreground picture of the 
first picture and the background picture of the second picture, 

the method comprising the steps of: 

detecting a change of pixel value between pixels constituting said first picture, and outputting change data 
expressing the distance on a color space between pixels close to each other of said first picture; 
generating said key signal on the basis of said change data integrated along a predetermined integral path in 
a direction in which a change of the pixel value has been detected by the detecting step; and 
synthesizing the foreground picture of the first picture and the background picture of the second picture by 
using the key signal generated by said generating step. 

15. A picture synthesis method as set forth in claim 14, 

the method comprising the step of 

detecting a difference between the foreground picture and the background picture of the first picture to output 
difference information indicating the difference. 

16. A picture synthesis method as set forth in claim 14 or 15, wherein said change data are integrated along said 
predetermined integral path within an integral range which is the range where said change data takes a predeter- 
mined threshold value or more to thereby generate the key signal. 



Patentansprtiche 

1. Schlussetsignal-Erzeugungsvorrichtung zur Erzeugung eines Schlusselsignals von einem Bild, 

wobei die Vorrichtung aufweist, 

eine Anderungs-Detektionseinrichtung zum Detektieren einer Pixelwertanderung zwischen das Bild darstel- 
lenden Pixeln und Ausgeben von den Abstand auf einem Farbraum zwischen zueinander nahen Pixeln des 
Bildes ausdruckenden Anderungsdaten, und 

eine Schlusselsignal-Erzeugungseinrichtung zur Erzeugung des Schlusselsignals auf der Basis der entlang 
einem vorbestimmten Integralpfad in einer Richtung, In der von der Anderungs-Detektionseinrichtung eine 
Anderung des Pixelwertes detektiert worden ist, integrierten Anderungsdaten. 

2. Schlusselsignal-Erzeugungseinrichtung nach Anspruch 1 , wobei die Schlusselsignal-Erzeugungseinrichtung zur 
Erzeugung des Schlusselsignals die Anderungsdaten entlang dem vorbestimmten integralpfad in einem integral- 
bereich integriert, welcher der Bereich ist, bei dem die Anderungsdaten einen vorbestimmten Schwellenwert oder 
mehr annehmen. 

3. Schlusselsignal-Erzeugungsvorrichtung nach Anspruch 1 Oder 2, wobei 

das Bild aus einem Vordergrundbild und einem Hintergrundbild besteht, und wobei 

die Vorrichtung auGerdem eine Differenz-Detektionsein richtung zum Detektieren einer Differenz zwischen 
dem Vordergrundbild und dem Hintergrundbild zum Ausgeben von diese Differenz anzeigenden Differenzda- 
ten aufweist. 

4. Schlusselsignal-Erzeugungsvorrichtung nach einem der Anspriiche 1 bis 3, wobei die Schlusselsignal-Erzeu- 
gungsvorrichtung aufweist: 

eine Normierungseinrichtung zur Normierung der Anderungsdaten durch die Differenzdaten zur Ausgabe von 
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als das Ergebnis der Normierung erhaltenen Normierungsdaten, und 

eine Integrations-Ausfuhrungseinrichtung zur Integration der nqrmierten Daten entlang dem vorbestimmten 
Integralpfad. 

5 5. Schlusselsignal-Erzeugungsvorrichtung nach einem der Anspruche 2 bis 4, wobei die Schlusselsignal-Erzeu- 
gungsvorrichtung aufweist: 

eine Bereich-Detektionselnrichtung zum Detektieren des Integraibereichs, 

eine Bestimmtintegralinformation-Erzeugungseinrichtung zur Integration der Anderungsdaten entlang dem 
10 vorbestimmten Integralpfad im Integralbereich zur Erzeugung einer Bestimmtintegralinformation der Ande- 

rungsdaten im Integralbereich, 

eine Normierungseinrichtung zur Normierung der Anderungsdaten durch die Bestimmtintegralinformation zur 
Ausgabe von als das Ergebnis der Normierung erhaltenen normierten Information, und 
eine Integrations-Ausfuhrungseinrichtung zur Integration der normierten Information, urn dadurch das Schliis- 
15 selsignal zu erzeugen. 

6. Zum Empfang eines aus wenigstens einem Vordergrund- und einem Hintergrundbild bestehenden ersten Bildes 
und eines aus wenigstens dem Hintergrundbild bestehenden zweiten Bildes ausgebildete Bildsynthesevorrichtung 
zum Synthetisieren des Vordergrundbildes des ersten Bildes und des Hintergrundbildes des zweiten Bildes, 

20 wobei die Vorrichtung aufweist: 

eine Anderungs-Detektionseinrichtung zum Detektieren einer Anderung eines Pixelwertes zwischen das erste 
Biid darstellenden Pixeln und Ausgeben von den Abstand eines Farbraumes zwischen zueinander nahen 
Pixeln des ersten Bildes ausdruckenden Anderungsdaten, 
25 eine Schlusselsignal-Erzeugungseinrichtung zur Erzeugung des Schlusselsignals auf der Basis der entlang 

dem vorbestimmten Integralpfad in einer Richtung, in der von der Anderungs-Detektionseinrichtung eine An- 
derung des Pixelwertes detektiert worden ist, integrierten Anderungsdaten, und 

eine Syntheseeinrichtung zum Synthetisieren des Vordergrundbildes des ersten Bildes und des Hintergrund- 
bildes des zweiten Bildes durch Verwendung des von der Schlusselsignal-Erzeugungseinrichtung erzeugten 
30 Schlusselsignals. 

7. Bildsynthesevorrichtung nach Anspruch 6, mit einer Differenz-Detektionsein richtung zum Detektieren einer Diffe- 
renz zwischen dem Vordergrundbild und dem Hintergrundbild des ersten Bildes zur Ausgabe einer die Differenzen 
anzeigenden Differenzinformation. 

35 

8. Bildsynthesevorrichtung nach Anspruch 6 oder 7, wobei die Schlusselsignal-Erzeugungseinrichtung die Ande- 
rungsdaten entlang dem vorbestimmten Integralpfad in einem Integralbereich integriert, welcher der Bereich ist, 
bei dem die Anderungsdaten einen vorbestimmten Schwellenwert oder mehr annehmen. 

40 9. Schlusselsignal-Erzeugungsverfahren zum Erzeugen eines Schlusselsignals von einem Bild, 
wobei das Verfahren die Schritte aufweist: 

Detektieren einer Anderung eines Pixelwertes zwischen das Bild darstellenden Pixeln und Ausgeben von 
Anderungsdaten, die den berechneten Abstand auf einem Farbraum zwischen zueinander nahen Pixeln des 
45 Bildes ausdrticken, und 

Erzeugen des Schlusselsignals auf der Basis der entlang einem vorbestimmten Integralpfad in einer Richtung, 
in der beim Detektionsschritt eine Anderung des Pixelwertes detektiert worden ist, integrierten Anderungsda- 
ten. 

so 10. Schlusselsignal-Erzeugungsverfahren nach Anspruch 9, wobei zur Erzeugung des Schlusselsignals die Ande- 
rungsdaten entlang dem vorbestimmten Integralpfad in einem Integralbereich integriert werden, welcher der Be- 
reich ist, bei dem die Anderungsdaten einen vorbestimmten Schwellenwert oder mehr annehmen. 

11. Schlusselsignal-Erzeugungsverfahren nach Anspruch 9 oder 10, 
55 wobei das Bild aus einem Vordergrundbild und einem Hintergrundbild besteht, wobei 

das Verfahren den Schritt eines 

Detektierens einer Differenz zwischen dem Vordergrundbild und dem Hintergrundbild zum Ausgeben von die- 
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se Differenz anzeigenden Differenzdaten aufweist. 

12. Schliisselsignalerzeugungsverfahren nach einem der Anspruche 9 bis 11, wobei das Verfahren die Schritte auf- 
weist: 

5 

Normieren der Anderungsdaten durch die Differenzdaten zum Ausgeben von als das Ergebnis der Norierung 
erhaltenen normierten Daten, und 

Integrieren der normierten Daten entlang dem vorbestimmten Integralpfad. 

10 13. Schlusselsignal-Erzeugungsverfahren nach einem der Anspruche 10 bis 12, wobei das Verfahren die Schritte 
aufweist: 

Detektieren des Integralbereichs, 

Integrieren der Anderungsdaten entlang dem vorbestimmten Integralpfad im Integralbereich, um Bestimmtin- 
15 tegralinformation der Anderungsdaten im Integralbereich zu erzeugen. 

Normieren der Anderungsdaten durch die Bestimmtintegral information zum Ausgeben der als das Ergebnis 
der Normierung erhaltenen normierten Information, und 

Integrieren der normierten Information, um dadurch das Schlusselsignal zu erhaften. 

20 14. Bildsyntheseverfahren zum Erhalten eines aus wenigstens einem Vordergrundbild und Hintergrundbild bestehen- 
den ersten Bildes und eines aus wenigstens einem Hintergrundbild bestehenden zweiten Bildes zum Synthetisie- 
ren des Vordergrundbildes des ersten Bildes und des Hintergrundbildes des zweiten Bildes, 
wobei das Verfahren die Schritte aufweist: 

25 Detektieren einer Anderung eines Pixelwertes zwischen das erste Bild darstellenden Pixeln und Ausgeben 

von den Abstand auf einem Farbraum zwischen zueinander nahen Pixeln des ersten Bildes ausdruckenden 
Anderungsdaten, 

Erzeugen des Schlusselsignals auf der Basis der entlang einem vorbestimmten Integralpfad in einer Richtung, 
in der beim Detektionsschritt eine Anderung des Pixelwertes detektiert worden ist, integrierten Anderungsda- 
30 ten, und 

Synthetisieren des Vordergrundbildes des ersten Bildes und des Hintergrundbildes des zweiten Bildes durch 
Verwendung des beim Erzeugungsschritt erzeugten Schlusselsignals. 

15. Bildsyntheseverfahren nach Anspruch 14, 
35 wobei das Verfahren den Schritt eines 

Detektierens einer Differenz zwischen dem Vordergrundbild und dem Hintergrundbild des ersten Bildes zum 
Ausgeben einer die Differenz anzeigenden Differenzinformation aufweist. 

40 16. Bildsyntheseverfahren nach Anspruch 1 4 oder 15, wobei zur Erzeugung des Schlusselsignals die Anderungsdaten 
entlang dem vorbestimmten Integralpfad in einem Integralbereich integriert werden, welcher der Bereich ist, bei 
dem die Anderungsdaten einen vorbestimmten Schwellenwert oder mehr annehmen. 



45 Revendications 

1. Un dispositif de generation de signal d'incrustation adapte pour gen6rer un signal d' incrustation a partir d'une 
image, 

le dispositif comprenant : 

50 

des moyens de detection de changement pour detecter un changement de valeur de pixel entre des pixels 
constituant I'image, et emettre des donnSes de changement exprimant la distance dans un espace de couleurs 
entre des pixels de I'image proches les uns des autres; et 

des moyens de generation de signal d'incrustation pourgen6rer le signal d'incrustation sur la base des don- 
55 nees de changement integrdes le long d'un chemin d'int6gration predetermine, dans une direction dans la- 

quelle un changement de la valeur de pixel a ete detecte par les moyens de detection de changement. 

2. Un dispositif de generation de signal d'incrustation seton la revendication 1 , dans lequel les moyens de generation 
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de signal cTincrustation integrant les donn6es de changement le long du chemin d'integration predetermine a 
I'interieur d'une plage d'integration qui est la plage dans laquelle les donn6es de changement prennent une valeur 
de seuil predeterminee, ou plus, pour generer ainsi le signal d'incrustation. 

5 3. Un dispositif de generation de signal d'incrustation selon la revendication 1 ou 2, dans lequel 

I'image consiste en une Image de premier plan et une image de fond, et dans lequel 
le dispositif comprend en outre des moyens de detection de difference pour detecter une difference entre 
I'image de premier plan et I'image de fond, pour foumir en sortie des donnees de difference indiquant cette 
10 difference. 

4. Un dispositif de generation de signal d'incrustation selon I'une quelconque des revendications 1 a 3, dans lequel 
les moyens de generation de signal d'incrustation comprennent : 

15 des moyens de normalisation pour normaliser les donnees de changement par les donnees de difference de 

facon a fournir en sortie des donnees normalisees obtenues comme le resultat de la normalisation, et 
des moyens d'execution d'integration pour integrer les donnees normalisees le long du chemin d'integration 
predetermine. 

20 5. Un dispositif de generation de signal d'incrustation selon I'une quelconque des revendications 2 a 4, dans lequel 
les moyens de generation de signal d'incrustation comprennent : 

des moyens de detection de plage pour detecter la plage d'integration; 

des moyens de generation d'information d'int6grale definie pour integrer les donn6es de changement le long 
25 du chemin d'integration predetermine a I'interieurde la plage d' integration, pour generer une information d'in- 

tegrale definie des donnees de changement a I'interieur de la plage d'integration; 

des moyens de normalisation pour normaliser les donn6es de changement par reformation d'int6grale definie, 
pour fournir en sortie une information normalisee obtenue comme le r6sultat de la normalisation, et 
des moyens d'execution d'integration pour integrer rinformation normalis6e pour g6n6rer ainsi le signal d'in- 
30 crustation. 

6. Un dispositif de synthese d'image adapte pour recevoir une premiere image consistent en au moins une Image 
de premier plan et une image de fond, et une seconde image consistant en au moins une image de fond, pour 
synthetiser I'image de premier plan de la premiere image et I'image de fond de la seconde image, 

35 le dispositif comprenant : 

des moyens de detection de changement pour detecter un changement de valeur de pixel entre des pixels 
constituant la premiere image, et pour fournir en sortie des donn6es de changement exprimant la distance 
dans un espace de couleurs entre des pixels de la premiere image proches les uns des autres; 

40 des moyens de generation de signal d'incrustation pour g6n6rer le signal d'incrustation sur la base des don- 

nees de changement int6grees le long d'un chemin d'integration predetermine dans une direction dans laquelle 
un changement de la valeur de pixel a 6te detecte par les moyens de detection de changement; et 
des moyens de synthese pour synthetiser I'image de premier plan de la premiere image et I'image de fond de 
la seconde image en utilisant le signal d'incrustation g6n6re par les moyens de generation de signal d'incrus- 

45 tation. 

7. Un dispositif de synthese d'image selon la revendication 6, comprenant en outre des moyens de detection de 
difference pour detecter une difference entre I'image de premier plan et I'image de fond de la premiere image, 
pour fournir en sortie une information de difference indiquant la difference. 

50 

8. Un dispositif de synthese d'image selon la revendication 6 ou 7, dans lequel les moyens de generation de signal 
d'incrustation integrent les donn6es de changement a I'interieur d'une plage d'integration qui est la plage dans 
laquelle {'information de changement prend une valeur de seuil pr6determinee, ou plus. 

55 9. Un procede de generation de signal d'incrustation pour generer un signal d'incrustation a partir d'une image, 
le procede comprenant les etapes suivantes : 

on detecte un changement de valeur de pixel entre des pixels constituant I'image, et on fournit en sortie des 
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donnees de changement exprimant la distance dans un espace de couleurs entre des pixels de I'image proches 
les uns des autres; et 

on genere le signal d'incrustation sur la base des donnees de changement integrees le long d'un chemin 
^integration predetermine dans une direction dans laquelle un changement de la valeur de pixel a et6 detecte 
5 par I'etape de detection. 

1 0. Un precede de generation de signal d'incrustation selon la revendication 9, dans lequel les donnees de changement 
sont integrees le long du chemin d'integration predetermine a I'interieur d'une plage d'integration qui est la plage 
dans laquelle les donnees de changement prennent une valeur de seuil predeterminee, ou plus, pour generer 

10 ainsi le signal d'incrustation. 

11. Un procede de generation de signal d'incrustation selon la revendication 9 ou 10, 

dans lequel I'image consiste en une image de premier plan et une image de fond, 
le procede comprenant I'etape consistant a 

15 

detecter une difference entre I'image de premier plan et I'image de fond pour fournir en sortie des donnees 
de difference indiquant cette difference. 

12. Un procede de generation de signal d'incrustation selon Tune quelconque des revendications 9 a 11, le procede 
20 comprenant les etapes suivantes : 

on normalise les donnees de changement par les donnees de difference pour fournir en sortie des donnees 

normalisees obtenues comme le resultat de la normalisation, et 

on integre les donnees normalisees le long du chemin d'integration predetermine. 

25 

13. Un procede de generation de signal d'incrustation selon I'une quelconque des revendications 1 0 a 12, dans lequel 
le procede comprend les etapes suivantes : 

on detecte la plage d'integration; 
30 on integre les donnees de changement le long du chemin d'integration predetermine a I'interieur de la plage 

d'integration, pour gen§rer une information d'integrale definie des donnees de changement a I'interieur de la 
plage d'integration; 

on normalise les donnees de changement par I'information d'integrale definie pour fournir en sortie une infor- 
mation normalisee obtenue comme le resultat de la normalisation; et 
35 on integre {'information normalis6e pour generer ainsi le signal d'incrustation. 

14. Un procede de synthese d'image dans lequel on recoit une premiere image consistant en au molns une image de 
premier plan et une image de fond, et une seconde image consistant en au moins une image de fond, pour syn- 
thetiser I'image de premier plan de la premiere image et I'image de fond de la seconde image, 

40 le precede comprenant les etapes suivantes : 

on detecte un changement de valeur de pixel entre des pixels constituant la premiere image, et on fournit en 
sortie des donn6es de changement exprimant la distance dans un espace de couleurs entre des pixels de la 
premiere image proches les uns des autres; 
45 on g6nere le signal d'incrustation sur la base des donn6es de changement integrees le long d'un chemin 

d'integration predetermine dans une direction dans laquelle un changement de la valeur de pixel a ete detecte 
par I'etape de detection; et 

on synthetise I'image de premier plan de la premiere image et I'image de fond de la seconde image en utilisant 
le signal d'incrustation g6n6r6 par I'etape de generation. 

50 

15. Un proc6de de synthese d'image selon la revendication 14, 

le procede comprenant I'etape suivante : 

on detecte une difference entre I'image de premier plan et I'image de fond de la premiere image pour fournir 
55 en sortie une information de difference indiquant la difference. 

16. Un procede de synthese d'image selon la revendication 14 ou 15, dans lequel les donnees de changement sont 
integrees le long du chemin d'integration predetermine a I'interieur de la plage ^integration qui est la plage dans 
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laquelle les donnees de changement prennent une valeur de seuil predetermine, ou plus, pour g6n6rer ainsi le 
signal d'incrustation. 



5 



10 



15 



20 



30 



35 



40 



45 



50 



30 



EP0 771 107 B1 



FOREGROUND BACKGROUND 




FIG.1A 



GRAY AREA 




FIG.1B 



31 



EP0 771 107 B1 




32 



EP0 771 107 B1 




FIG.3A 



33 



EP0 771 107 B1 



LOCAL AREA 



TEXTURE 




FIG.4 



34 



EP0 771 107 B1 




35 



EP0 771 107 B1 




36 



EP0 771 107 B1 




37 



EP0 771 107 B1 




EP0 771 107 B1 




39 



EP0 771 107 B1 



3 



< 

z 

o 

5) 
& 

_L 

w£</> 
*5 




40 



EP0 771 107 B1 



( START ) 



CALCULATION OF COLOR CHANGE C OF REMARKED 
PIXEL AND ADJACENT PIXEL OF INPUT PICTURE 




r 


DETECTION OF FE/I 
PATH DETERMINATK 


JURE POINT AND 
}N CLOSED CURVE 



S1 
S2 

r 



S3 



CALCULATION OF COLOR CHANGE F-B OF FOREGROUND 
AND BACKGROUND AT REMARKED PIXEL OF INPUT PICTURE 



S4 



NORMALIZATION OF COLOR CHANGE C BY COLOR 
CHANGE IF-BI TO CALCULATE NORMALIZED VALUE 

C 



I F-B I 



S5 



CALCULATION OF SKELETON PORTION OF KEY SIGNAL 
BY DEFINITE INTEGRAL OF NORMALIZED VALUE 

C 
IF-BI 



■S6 



GENERATION OF KEY SIGNAL 
CORRESPONDING TO INPUT PICTURE 



C ) 



FIG.11 



41 



EP0 771 107 B1 



r" 


n 


r 12R 


R 


r 13 




1-ST ORDER DIFFERENTIAL 
VALUE CALCULATING SECTION 








r 12G 






RGB 
DECOMPOSITION 
SECTION 


G 


1-ST ORDER DIFFERENTIAL 
VALUE CALCULATING SECTION 


G 


VECTOR 
SYNTHESIS 
SECTION 






r 12B 








B 


1-ST ORDER DIFFERENTIAL 
VALUE CALCULATING SECTION 


B' 





10 

FIG.12 



Q START ^ 



S1-1 



DECOMPOSITION OF PICTRUE C INTO 
COMPONENTS OF RESPECTIVE AXES 
DIRECTIONS (R, G, B) OF COLOR SPACE 



S1-2 



CALCULATION OF 1-ST ORDER DIFFERENTIAL 
VALUES WITH RESPECT TO RESPECTIVE 
COMPONENTS (R, G, B) TO DETERMINE 
DIRECTIONS AND MAGNITUDES OF CHANGES 
OF RESPECTIVE COMPONENTS 



r 



S1-3 



SYNTHESIS OF DIRECTIONS AND MAGNITUDES 

OF RESPECTIVE COMPONENTS (R, G, B) 
TO DETERMINE CHAGE C (VECTOR^ OF PIXEL 



Q END ~) 



FIG.13 



42 



EP0 771 107 B1 



SOx 



-1 


0 


1 




-1 


-2 


-1 


-2 


0 


2 


SOy 


0 


0 


0 


-1 


0 


1 




1 


2 


1 



|Sf| = V fx^ + fy 2 
e = tan" 1 (fy/fx) 



FIG.14 




FIG.15 



43 



EP 0 771 107 B1 




o> 

lis 

So 



o> 

j=tE 

lis 



Ul 



CO 



So 



8 



CO 

d 



CM 2kuj 
(DO 



(l Oujw 



I 



Ui 

goo 

Is* 



44 



EP0 771 107 B1 



C 




45 



EP0 771 107 B1 



C START ) 





r S2-1 


DETECTION OF BOUNDARY OF OBJECT 
OF PICTURE C ON THE BASIS OF 
COLOR CHANGE C OF PIXEL VALUE 






r S2-2 


DITECTION OF FEATURE POINT 











C a* 0 ) 

FIG.18 



EP0 771 107 B1 




47 



EP0 771 107 B1 



VERY SMALL 
AREA 

/ 



FEATURE 
POINT 



FIG.20A 



DISTANCE , 
CORRESPONDING / 
TO 4 PIXELS 




PATH 
DETERMINATION 
CLOSED CURVE 



PATH 
s DETERMINATION 
\ CLOSED CURVE 

\ 



FIG.20B 




FEATURE 
POINT 



48 



EP0 771 107 B1 



COLOR 




FIG.21 



49 



EP0 771 107 B1 



Q START ) 



DETECTION OF 
VERY SMALL AREA 



r 



S3-1 



S3-2 



FORMATION OF COLOR DISTRIBUTIONS 
OF FOREGROUND F AND BACKGROUND B 



r 



S3>3 



DETECTION OF RESPECTIVE CENTERS OF 
GRAVITY OF COLOR DISTRIBUTIONS OF 
FOREGROUND F AND BACKGROUND B 



JL 



S3-4 



CALCULATION OF DIFFERENCE BETWEEN 
CENTER OF GRAVITY OF FOREGOUND F AND 
CENTER OF GRAVITY OF BACKGROUND B 



S3-5 



CALCULATION 
WITH RESPECT TO ALL VERY 
SMALL AREAS COMPLETED 

? 



YES 



END 



3 



FIG.22 



50 



EP0 771 107 B1 




FIG.23 



51 



EP0 771 107 B1 



3 



UJUI 

>> 
cccc 

oo 

DO 
UJUI 

33 

oo 

_L 

o 



Q 



2g 

si 




uj<oco 

(.CLLUUJ 
Q 



T 



CO 



9 



o> Z 
qO u. 



<JU2 

^2 



SuilU 



HI 



IP 



I 



0CD W 



CM 

S3 

li- 



es 
z 

"Poo 

-si 

O 

£ 



52 



EP0 771 107 B1 




FIG.25A 



P2 CLOSED 
CURVE 



P1 CLOSED 
CURVE 

i 

/ 




PATH 
DETERMINATION 
CLOSED CURVE 



FIG.25B 



53 



EP0 771 107 B1 



C START ) 



S4-1 



DETERMINATION OF 
INTEGRAL PATH 



\ r S 4-2 

NORMALIZATION OF 
C*BY|F-B| 




CONNECTION OF RESPECTIVE 
POINTS OF as 0, a=1 TO 

GENERATE CLOSED CURVE OF 
a s 0(P1 CLOSED CURVE) 
AND CLOSED CURVE OF 
a = 1(P2 CLOSED CURVE) 



( END ) 



FIG.26 



54 



EP0 771 107 B1 



51 



a o- 



52 > 



P1 CLOSED 
CURVE o- 

P2 CLOSED 
CURVE 



INTERPOLATING 

SECTION 
FOR INSIDE OF 
GRAY AREA 



0/1 

ALLOCATING 
SECTION 



r 



54 



SYNTHESIS 
SECTION 



KEY 
SIGNAL 



53 



INTERPOLATING 
SECTION 
FOR AREA EXCEPT 
FOR GRAY AREA 



50 



FIG-27 



55 



EP0 771 107 B1 




PICTURE 
FRAME PATCH 
AREA 



B 



PATH 
SEGMENT 




,P2 CLOSED 
CURVE 



P1 CLOSED 
CURVE 



PATH 
DETERMINATION 
CLOSED CURVE 



FIG.28B 



56 



EP0 771 107 B1 



( start ) 



ALLOCATION OF 1,0 
ONTO P1.P2 CLOSED CURVES 



S5-1 



GENERATION OF INTERPOLATED 
CURVED SURFACE Q 



l/ 



S5-2 



GENERATION OF KEY SIGNAL 
WITHIN PATCH AREA 



l/ 



S5>3 



INTERPOLATION OF 1 INTO 
AREA SURROUNDED BY PI CLOSED CURVE 
AND 0 INTO AREA SURROUNDED BY P2 
CLOSED CURVE AND PICTURE FRAME 



V 



S5-4 



SYNTHESIS 



l/ 



S5-5 



^ END ^ 



FIG.29 



57 



EP0 771 107 B1 



PICTURE 

C O- 



30 



AO 



CHANGE 
DETECTING 
SECTION 



c 

INTEGRAL 
VALUE 
CALCULATING 
SECTION 



■50 



F-B 
DETECTING 
SECTION 



OPERATION 
SECTION 



KEY SIGNAL 
GENERATING 
SECTION 



, KEY 
SIGNAL a 



FEATURE POINT 

PATH DETERMINATION 
CLOSED CURVE 

■60 



FEATURE POINT 
DETECTING SECTION 



FIG.30 



OPERATION 
SIGNAL 




PATH 
DETERMINATION 
CLOSED CURVE 



22 



FEATURE POINT 
DETERMINING 
SECTION 



FEATURE 
o POINT 



60 



PATH 
^ DETERMINATION 
*° CLOSED CURVE 



FIG.31 



58 



EP0 771 107 B1 



FIG.32A 



PIXEL 



FIG.32B 




PATH DETERMINATION 
CLOSED CURVE . 



PIXEL 



10 



JO 



,50 



PICTURE 

C o— 



CHANGE 
DETECTING 
SECTION 



20 



INTEGRAL 
VALUE 
CALCULATING 
SECTION 



FEATURE 
POINT 
DETECTING 
SECTION 



KEY 
SIGNAL 
GENERATING 
SECTION 



KEY 
SIGNAL 



a 



FIG.33 



59 



EP0 771 107 B1 



FIG.34A 




KEY SIGNAL 



FIG.34B 




p 2 POSITION p 



60 



EP0 771 107 B1 



( START ) 





r 


CALCULATION OF COLOR CHANGE C 

OF REMARKED PIXEL AND 
ADJACENT PIXEL OF INPUT PICTURE 




: £i 


DETECTION OF FE> 
PATH DETERMINATH 


VTURE POINT AND 
ON CLOSED CURVE 



S21 



$22 



S23 



DETERMINATION OF POINTS Pi i, P2 WHERE MAGNITUDE 
OF CHANGE C OF PIXEL VALUE BECOMES EQUAL TO K 
TO CALCULATE INTEGRAL VALUE OF CHANGE C OF 
PIXEL VALUE C WITHIN THAT RANGE 



NORMALIZATION OF COLOR CHANGE C BY 



iT 



D TO CALCULATE NORMALIZED VALUE 



S24 



S25 



EXECUTION OF INTEGRATION OF 
P C'(p) 

P1 



r 

J I 



dp TO 



CALCULATE SKELETON PORTION OF KEY SIGNAL 



GENERATION OF KEY SIGNAL a 
CORRESPONDING TO INPUT PICTURE 



S26 



C ENP ) 
FIG.35 



61 



EP0 771 107 B1 



i 



ill 
pi 



O Q 




S<ujPS 

U.UJ<30 
UJH>OUI 
QZ JO) 

O 



o 

2x5o 




D Q 

_L_ 



o 

HP 

ills 

W UJ 

a 



6 
b 



8 



CD 
CO 

d 



62 



EP0 771 107 B1 



C start ) 



J 



S21-1 



DETERMINATION OF INTEGRAL PATH 



S21-2 



DETECTION OF POINT Pi (POINT OF ce=0), 
POINT P2 (POINT OF <x=1) WHERE MAGNITUDE OF C ON 
INTEGRAL PATH BECOMES EQUAL TO K 






INTEGRATION OF C EX 
FROM POINT Pi TO POI* 
PATH TO CALCULATE DEF 


ISTING WITHIN RANGE 
fT P2 ALONG INTEGRAL 
INITE INTEGRAL VALUE D 



S21-3 



NO 



NORMALIZATION OF C BY D 



INTEGRATION OF jp ALONG INTEGRAL 
PATH TO DETERMINE a 



if 



S21-4 



S21-5 




INTEGRATION ALONG 
ALL INTEGRAL PATHS 
PASSING THROUGH 
FEATURE POINTS COMPLETED ? 




S21-6 



YES 



GENERATION OF PI CLOSED CURVE, 
P2 CLOSED CURVE 



1/ 



S21-7 



( ) 



FIG.37 



63 



EP0 771 107 B1 




00 
CO 

(5 

UL 



64 



EP0 771 107 B1 




CO 

6 



65 



EP0 771 107 B1 



O) 



i 



5' 



8- 



o 

d 

LL 



o 

IUZZ 

opo 
Zot= 
<uio 

XHUJ 
OUJOT 

o 



66 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 



□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 




FADED TEXT OR DRAWING 



